Semi Senior QA Engineer – AI Applications

About the role

Rootstrap is building a pipeline for future AI‑focused QA talent. The Semi Senior QA Engineer will test non‑deterministic AI systems such as large language models, AI agents, and Retrieval‑Augmented Generation (RAG) applications, while contributing to evaluation frameworks that ensure quality and detect regressions.

Key responsibilities

• Create comprehensive test plans, test cases and automation scripts (Playwright‑based).
• Review client requirements and technical specs with developers, providing timely feedback.
• Estimate, prioritize, and coordinate testing activities, tracking bugs and quality metrics.
• Design, develop and execute regression tests, including automation integration.
• Build and maintain AI evaluation suites to validate model behavior and detect hallucinations, grounding issues, etc.
• Analyze AI outputs, identify failure patterns, and collaborate with engineers on preventive solutions.

Required profile

• 3+ years of experience as a QA Engineer with mobile testing expertise.
• Advanced English proficiency and a background in Software Engineering or Computer Science.
• Hands‑on experience testing AI‑powered applications and evaluating non‑deterministic systems.

Required skills

• Playwright test automation.
• Mobile testing tools and frameworks.
• AI tools such as Claude for test case generation and bug reporting.
• Evaluation frameworks (LangSmith, Braintrust, Arize Phoenix, Promptfoo, OpenEvals) and benchmark dataset creation.
• CI/CD pipeline integration for automated testing.

What we offer

• Remote work flexibility with optional office presence in Montevideo, Buenos Aires, or Medellín.
• Professional development through certifications, mentorship, and learning bonuses.
• Well‑being benefits including gym access, psychological counseling, and weekly lunch reimbursements.
• On‑boarding kit, cutting‑edge tools, and a workation program.