Multilingual Audio Personalization Evaluator - English (United States)

About the Role  We are looking for experienced annotators with deep cultural, linguistic, and audio catalog expertise to help improve a major music platform’s personalized experiences across multiple languages and regions.  In this role, you will create high-quality ground truth and evaluate the quality of both training data and model-generated output for various features. Your insights will help shape how listeners around the world discover and enjoy music, podcasts, and audiobooks.  What You’ll Do  - Evaluate agentic and personalized experiences and assess search relevance & recommendations  - Perform complex qualitative assessments to support emerging agentic and conversational listening experiences.  - Judge how well recommendations, responses, and UI elements reflect local culture, language norms, and audio habits.  - Review playlist and recommendation quality, including how well results match inferred user intent and preferences.  - Assess how realistically synthetic prompts/queries represent real user behavior and catalog usage in your region.  - Create detailed ground truth judgments for music, podcasts, and audiobooks.  - Validate and refine LLM-generated judgments to ensure they align with real-world user expectations and regional norms.  - Identify inconsistencies, edge cases, and nuanced failures in model output.  - Summarize insights & highlight risks to provide concise written and verbal summaries of your top findings, highlights, and concerns for each evaluation set.  - Carefully review and follow project-specific instructions before starting annotation work.  - Adapt quickly as project types rotate and priorities shift and deliver high-quality annotations within 1–2 week deadlines.  Requirements - Native proficiency in English  with deep understanding of the regional culture associated with your language, including how people in your region actually use audio streaming services.  - Extensive, current knowledge of music, podcasts, and audiobooks relevant to your region such as their key artists, global trends, listening habits. - Long-time, avid fan of music, with additional familiarity using streaming experiences for podcasts and audiobooks.  - Could be an avid listener, musician, or songwriter; formal industry experience is NOT required. - 6+ months of annotation or labeling experience, ideally in complex or subjective domains.  - Critical thinking and strong research skills to supplement your audio content knowledge. -Proficient with Google Workspace, especially Google Sheets, for tracking annotations and evaluations with great accuracy. - Open to learning and feedback with an ability to meet deadlines for 1–2 week annotation cycles. Project Details Contract Type: Freelance  Location: United States (remote) Start Date : 15th December 2025 Duration: 1 to 2 weeks Schedule: Daily 1 hour; flexible based on client's needs Note: Please do not use VPNs or IP-masking tools during the recruitment process — our security system requires accurate regional verification.