ML Engineer – Agentic Systems Validation

ML Engineer – Agentic Systems Validation (0–2 Years Experience)

Location: Pune, India

Job Profile: QATGP3

Role Overview

We’re looking for an early-career SDET (Software Development Engineer in Test) to help build and maintain quality practices for AI-powered and agentic systems—including LLM-based features, tool-using agents, and automated workflows.

You’ll work embedded within the ML/AI team, collaborating closely with ML engineers, software engineers, and product partners to ensure AI experiences are reliable, testable, and production-ready. This role is a great fit for someone who enjoys learning fast, writing automation, and validating complex behaviors across modern systems—especially systems that can behave differently run-to-run due to non-determinism.

Key Responsibilities

End-to-End Testing for Agentic Systems

• Execute and expand end-to-end testing for complex AI and multi-agent workflows

• Help validate agent behaviors such as:

  • multi-step execution and task completion

  • tool/API calling correctness

  • context handling and state transitions

  • retry/fallback behavior under failures

• Assist in building and maintaining regression test coverage using golden datasets and repeatable evaluation scenarios

• Create and maintain test cases for functional, integration, and regression testing across services and UI

Test Automation & Quality Engineering

• Contribute to and maintain test automation frameworks for API and service-level testing (and UI where needed)

• Write automated tests as part of CI/CD pipelines to provide fast feedback for engineering teams

• Investigate failures and help triage issues with engineers using logs, error traces, and test reports

• Improve test data setup, test environments, and test execution reliability

Production Monitoring & Support

• Assist with monitoring agentic systems in production, tracking:

  • workflow failures

  • unexpected agent behavior or degraded quality

  • latency and dependency issues

• Help analyze incidents, reproduce problems, and contribute to post-incident fixes and regression prevention

• Support observability efforts by validating key signals (logs/traces/metrics) and contributing to dashboards

Collaboration & Learning

• Work closely with the ML/AI team to understand system behavior and quality risks

• Participate in code reviews, test reviews, and sprint planning as a quality partner

• Learn AI testing patterns and contribute ideas to improve reliability and evaluation practices

Required Skills

• Basic to intermediate programming skills in Python, Java, or JavaScript/TypeScript

• Familiarity with automated testing concepts: unit tests vs integration tests vs E2E tests

• Some experience with at least one testing framework (examples):

  • Python: pytest, unittest

  • Java: JUnit/TestNG

  • JS/TS: Jest, Playwright

• Understanding of APIs and service testing (REST fundamentals, JSON, HTTP)

• Comfort debugging issues using logs, error traces, and test reports

• Good understanding of non-deterministic agentic systems, including:

  • why AI/LLM-based behaviors can vary between runs

  • how multi-step agent workflows are developed (planning → tool use → observation → iteration)

  • practical approaches to testing agentic workflows (golden tests, controlled inputs, mocking tools/dependencies, replayable scenarios, and evaluation-based assertions rather than exact string matching)

• Strong communication skills and eagerness to collaborate in a fast-moving team

Qualifications

• 0–2 years of experience in QA automation, SDET, software engineering, internships, co-ops, or academic projects

• Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience

• Exposure to CI/CD tools (GitHub Actions, Jenkins, GitLab CI, etc.) is a plus

• Interest in AI/ML systems and motivation to grow into AI quality engineering

Nice-to-Have

• Personal projects or coursework involving AI/ML, chatbots, LLMs, or workflow automation

• Exposure to Playwright/Selenium or API testing tools (Postman, REST Assured, etc.)

• Familiarity with cloud basics (AWS/GCP/Azure) and microservice concepts

• Interest in observability tools (Datadog, Grafana, ELK, OpenTelemetry)

• Awareness of AI safety concepts (prompt injection, data leakage risks)