Freelance AI Agent Evaluation Engineer

About the role

Mindrift is looking for a freelance engineer to help build a high‑quality dataset for evaluating AI coding agents. You will design realistic developer environments and craft challenging tasks that test the limits of current models.

Key responsibilities

• Construct virtual company environments with codebases, infrastructure, tickets, documentation, and conversation histories.
• Design tasks from intermediate states, write clear prompts, and define objective success criteria.
• Develop functional and integration tests that accept all correct solutions while rejecting incorrect ones.
• Iterate on tasks and tests based on QA feedback, analysing agent failures and refining evaluation criteria.

Required profile

• 5+ years of professional software development experience.
• Strong command of English (minimum B2 level).
• Hands‑on experience writing functional and integration tests.

Required skills

• Python (FastAPI)
• JavaScript / TypeScript (React)
• Docker
• PostgreSQL
• Kafka
• Redis

What we offer

• Project‑based freelance contract with flexible hours.
• Compensation up to $40 USD per hour, based on experience and pace.
• Remote work – you choose when and how to complete the tasks.