Member of Technical Staff, Model Evaluation

Location: Bay Area

Department: Research

Location Type: IN_OFFICE

Employment Type: FULL_TIME

• Design, develop, and maintain robust evaluation frameworks and benchmarks for measuring LLM performance across diverse tasks and domains.
• Define and implement quantitative metrics that capture model quality, safety, reliability, and regression detection.
• Build scalable, automated evaluation pipelines that integrate into model training and deployment workflows.
• Conduct rigorous statistical analysis of model outputs to identify failure modes, biases, and performance gaps.
• Partner with product and customer-facing teams to translate real-world use cases into meaningful evaluation criteria.

• BS/MS/PhD in Computer Science, Machine Learning, Statistics, or a related field (or equivalent experience).
• At least 2 years of experience in ML evaluation, applied ML research, or a related engineering role.
• Strong understanding of LLM fundamentals (autoregressive generation, instruction tuning, RLHF, in-context learning, decoding strategies).
• Proficiency in Python and ML frameworks such as PyTorch.
• Experience designing and implementing evaluation metrics and benchmarks for generative models.
• Solid foundation in statistics, experimental design, and hypothesis testing.
• Experience with version control (Git) and containerization (Docker).
• Excellent communication skills with the ability to distill complex evaluation results into actionable insights.

• Experience with human-in-the-loop evaluation systems (Likert-scale annotation, pairwise preference ranking, red-teaming).
• Familiarity with LLM safety and alignment evaluation (toxicity, hallucination detection, factual grounding).
• Knowledge of existing benchmark suites (MMLU, HumanEval, HELM, BIG-Bench) and their limitations.
• Experience building evaluation infrastructure at scale using cloud platforms (AWS, GCP, Azure).
• Familiarity with MLOps practices and CI/CD pipelines for model validation.
• Experience with data engineering, large-scale data labeling, or synthetic data generation for evaluation purposes.