AI Evaluation SDET – Remote

About the role

We are looking for an experienced Software Development Engineer in Test (SDET) to design and execute evaluation frameworks for AI agents that generate code. The role is fully remote and focuses on ensuring AI model outputs are correct, robust, and reliable.

Key responsibilities

• Design verifiers and correctness rubrics for coding tasks to validate AI‑generated code.
• Identify edge cases and create adversarial test scenarios for comprehensive model evaluation.
• Grade agent trajectories and continuously improve test quality through detailed reviews.
• Work independently and asynchronously to meet deadlines while enhancing AI model performance.
• Collaborate with subject‑matter experts to maintain test consistency and relevance.

Required profile

• 5+ years of experience as an SDET or software test engineer in a product‑focused organization.
• Strong written communication skills.
• Experience with CI/CD pipelines.
• Familiarity with AI tools and evaluation processes is a plus.

Required skills

• Automation frameworks such as pytest, Playwright, Cypress.
• CI/CD processes.

What we offer

• Hourly rate ranging from $30 to $100.
• 30+ hours per week on a remote contract basis.
• Weekly payments via Stripe Connect.