Senior Manager – GenAI Performance & Analytics

AI Performance Analytics

• Analyze large volumes of chatbot and AI conversation data to identify performance gaps and improvement opportunities.

• Define and track key AI performance metrics such as resolution rate, containment rate, response quality, and customer satisfaction.

• Build dashboards and reporting frameworks using SQL, Python, and BI tools such as Tableau.

LLM Evaluation & Quality Insights

• Design frameworks to evaluate the response quality of LLM responses across dimensions such as:

  • factuality and groundedness

  • Relevance to user intent

  • Instruction Adherence

  • tone and customer experience

  • safety and compliance

  • Build scalable processes for LLM-based automated evaluation and human review workflows.

  • Identify systematic failure patterns in AI responses and recommend improvements.

  Model Improvement & Experimentation

  • Partner with AI and engineering teams to evaluate model upgrades, prompt changes, and new AI capabilities.

  • Design and analyze experiments to measure the impact of model improvements.

  • Generate insights that inform model training data, prompts, and product features.

  Conversational Insights

  • Identify trends in customer interactions to uncover:

    • new AI automation opportunities

    • gaps in intent coverage

    • common customer pain points

    • Translate analytics insights into actionable recommendations for product and AI teams.

    Team Leadership

    • Lead and mentor a team of data analysts focused on AI performance and conversational analytics.

    • Establish best practices for analytics, experimentation, and AI evaluation.

    • Collaborate closely with product, AI research, engineering, and customer success teams.

Experience

• 7–10 years of experience in data analytics, data science, or AI analytics.

• Experience managing or mentoring analytics teams.

• Experience analyzing conversational data, customer support data, or AI-driven products is a strong plus.

Technical Skills

• Strong SQL and data analysis skills.

• Experience with BI tools such as Tableau or similar platforms.

• Experience using Python for data analysis or experimentation.

• Familiarity with machine learning or large language models (LLMs).

Preferred Experience

• Experience evaluating AI or machine learning model performance.

• Experience working with conversational AI platforms or chatbots.

• Experience with experimentation frameworks and A/B testing.