Research Scientist – Frontier Model Evaluation (Remote)

About the role

We are looking for a Research Scientist to own how Anyone AI measures frontier model capability. This fully remote position will design, build, and validate evaluation benchmarks for large language models across reasoning, coding, agents, tool use, and multi‑modal tasks.

Key responsibilities

• Design original evaluation frameworks and turn public benchmarks into proprietary evaluation packages.
• Address construct validity, discrimination, headroom, and contamination in benchmark development.
• Build evaluation packages with expert‑verified ground truth, multi‑model headroom results, and rigorous quality control.
• Recruit, calibrate, and oversee a pool of subject‑matter experts in coding, agentic/tool‑use, and STEM reasoning.
• Serve as technical liaison with partner labs, translating measurement goals into evaluation designs.
• Deliver pilot evaluation packages end‑to‑end, ensuring they are lab‑ready before release.

Required profile

• Research background in machine‑learning evaluation or benchmarking, with published or open benchmarks.
• Deep expertise in large language model benchmarking, especially code‑model evaluation.
• Fluency in frontier model measurement concepts such as rubrics, pass rates, headroom, and contamination.
• Proven ability to maintain rigorous standards across expert pools.
• Fluent English; Spanish is a plus.

Required skills

• Machine Learning evaluation
• Large Language Model benchmarking
• Evaluation design and validation
• Statistical analysis of benchmark results
• Data annotation and quality control processes