Freelance Agent Evaluation Engineer

About the role

Mindrift is building a dataset to evaluate AI coding agents on realistic developer tasks. As a Freelance Agent Evaluation Engineer you will design challenging tasks, create evaluation criteria, and ensure the assessment is fair and robust.

Key responsibilities

• Construct realistic developer environments that include codebases, infrastructure, tickets, documentation, and conversation history.
• Design tasks from intermediate states of these environments, write clear prompts, and define success criteria.
• Develop functional and integration tests that accept all correct solutions while rejecting incorrect ones.
• Iterate on tasks and tests based on QA feedback, reviewing agent solutions and refining evaluation logic.

Required profile

• 5+ years of software development experience.
• Strong command of English (minimum B2 level).
• Hands‑on experience with Python (FastAPI), JavaScript/TypeScript (React), Docker, PostgreSQL, Kafka and Redis.
• Proven ability to write comprehensive functional and integration tests.

Required skills

• Python
• FastAPI
• JavaScript
• TypeScript
• React
• Docker
• PostgreSQL
• Kafka
• Redis
• Functional testing
• Integration testing

What we offer

• Up to $40 USD per hour, based on experience and pace.
• Fully remote, flexible schedule – you choose when and how to work.
• Project‑based freelance engagement with leading AI companies.