Freelance AI Agent Evaluation Engineer

About the role

We are looking for a Freelance AI Agent Evaluation Engineer to help build a high‑quality dataset that assesses how AI coding agents handle real‑world developer tasks. The role is project‑based, fully remote, and offers flexible hours.

Key responsibilities

• Construct realistic developer environments, including codebases, infrastructure, tickets, documentation and conversation histories.
• Design evaluation tasks from intermediate states of these environments, define success criteria and ensure they are solvable by an AI agent.
• Write functional and integration tests that accept all correct solutions while rejecting incorrect ones.
• Iterate on tasks and tests based on QA feedback, analysing agent failures and refining evaluations for fairness and robustness.

Required profile

• 5+ years of professional software development experience.
• Strong command of English (minimum B2 level).
• Hands‑on experience with Python (FastAPI) and JavaScript/TypeScript (React).
• Familiarity with Docker, PostgreSQL, Kafka and Redis.
• Proven ability to write comprehensive functional and integration tests.

Required skills

• Python
• FastAPI
• JavaScript
• TypeScript
• React
• Docker
• PostgreSQL
• Kafka
• Redis
• Functional testing
• Integration testing

What we offer

• Project‑based freelance contract with compensation up to $40 per hour.
• Fully remote work with the freedom to set your own schedule.
• Opportunity to influence cutting‑edge AI evaluation datasets for leading tech companies.