Human Evaluation – Program Manager
Our team plays a critical role in ensuring the quality and integrity of machine learning models. We are responsible for designing, implementing, and analyzing human evaluation studies to assess model performance and provide actionable insights. We collaborate closely with researchers, engineers, and product managers across various teams to advance ML capabilities.
Responsibilities:
- Define and manage comprehensive human evaluation programs for various machine learning models and applications.
- Collaborate with ML engineers and researchers to translate complex research questions into scalable human evaluation tasks and protocols.
- Design and implement detailed data collection methodologies, including survey design, task creation, and rater training materials.
- Monitor evaluation progress, meticulously analyze results, and generate clear, actionable insights and recommendations.
- Communicate findings, insights, and recommendations effectively to diverse stakeholders across the organization, including executive leadership.
- Manage and optimize a pool of external evaluators, ensuring high-quality data collection and adherence to guidelines.
- Continuously identify opportunities to improve evaluation processes, tools, and methodologies to enhance efficiency and impact.
Prerequisites/Qualifications:
- 5+ years of experience in program management, project management, or a related field, with a strong focus on human evaluation, data labeling, or ML data pipelines.
- Demonstrated expertise in designing and executing large-scale human evaluation studies, including experimental design and statistical analysis.
- Solid understanding of machine learning concepts, methodologies, and their practical application in real-world scenarios.
- Exceptional analytical and problem-solving skills, with the ability to interpret complex data, identify trends, and draw robust conclusions.
- Proficiency in data analysis tools (e.g., Python, R, SQL) and statistical methods.
- Outstanding communication (written and verbal) and interpersonal skills, with the ability to collaborate effectively with diverse technical and non-technical teams.
- Proven ability to manage multiple concurrent projects, prioritize tasks, and thrive in a fast-paced, dynamic environment.
- Bachelor’s or Master’s degree in a quantitative field (e.g., Computer Science, Statistics, Psychology, Data Science) or equivalent practical experience.
