Staff Data Scientist

Job Overview:

Our team is responsible for ensuring that AI systems deployed at scale are measurable, trustworthy, and decision‑ready. We build rigorous evaluation frameworks, analytics platforms, and KPI audits that directly influence product direction and real‑world safety outcomes. This role is primarily evaluation and analytics driven. As a Staff Data Scientist will own the definition, execution, and evolution of evaluation frameworks, metrics, and analytical methodologies used to assess AI/ML feature performance in real‑world deployments. Rather than focusing on core model development, this role emphasizes measurement rigor, error analysis, experimentation, and decision support—ensuring that metrics accurately reflect system behavior, business impact, and safety outcomes.

Key Responsibilities:​

Evaluation & Measurement Ownership

• Design, implement, and maintain offline and online evaluation frameworks for AI/ML features.
• Define, validate, and evolve KPIs, success metrics, and audit methodologies used across teams.
• Perform deep error analysis, bias analysis, and segmentation to identify failure modes and improvement opportunities.
• Own golden datasets, validation protocols, and benchmarking standards.

Analytics & Insight Generation

• Conduct large‑scale analytical studies to understand feature performance, data quality issues, and system behavior.
• Translate complex analytical findings into clear, actionable insights for engineering, product, and leadership stakeholders.
• Challenge existing metrics or evaluation approaches when they fail to capture ground reality.

Experimentation & Statistical Rigor

• Design and review experiments including offline evaluations, controlled rollouts, and A/B tests.
• Ensure statistical correctness in analysis, including bias, variance, confidence intervals, and significance.
• Perform post‑deployment monitoring and regression detection.

Tooling, Automation & Scale

• Build and maintain tools, dashboards, and automation frameworks to scale audits, evaluations, and reporting.
• Improve repeatability, reproducibility, and reliability of analytics pipelines.
• Enable self‑serve analytics and standardized reporting for broader teams.

Leadership & Ownership

• Independently identify gaps in evaluation, metrics, or data quality and drive solutions end‑to‑
• Mentor junior data scientists on statistical rigor, experiment design, and analytical storytelling.

Mandatory Skills:​

• Tech, M.Tech, or PhD in Computer Science, Electrical Engineering, Statistics, Mathematics, or a related field.
• 5+ years of experience in data science, analytics, or a closely related domain.
• Strong foundation in probability, statistics, and estimation theory.
• Strong analytical and problem‑solving skills with keen attention to detail.
• Strong programming skills in Python, with solid fundamentals in OOP, algorithms, and data structures.
• Deep familiarity with SQL, complex query writing, indexing, and database internals; working knowledge of at least one NoSQL data store.
• Experience with data visualization and analytical storytelling.
• Excellent written and verbal communication skills.
• Familiarity with AI‑powered tools for analytics and software development, including:
  • Using AI tools for exploratory data analysis, feature ideation, experiment analysis, and documentation.
  • Leveraging AI assistance for rapid prototyping, code refactoring, debugging, and analytical workflows.
  • Ability to critically evaluate AI‑generated outputs for correctness, statistical validity, reproducibility, and production readiness.

 Preferred Skills:​

• Exposure to cloud platforms and services (e.g., AWS Kinesis, EKS, autoscaling systems).
• Experience building lightweight web or service components using Python frameworks (e.g., Flask, Django).
• Prior experience working with large‑scale, noisy, real‑world datasets