Senior Data Scientist AI Evaluation

Accountabilities: AI Evaluation Design: Define ground truth, quality metrics, scoring methodologies, and evaluation frameworks to assess the accuracy, consistency, and reliability of AI models and agents. Evaluation Pipeline Development: Build repeatable evaluation loops that monitor model quality over time, identify performance regressions, and support reliable release decisions. Statistical Measurement and Validation: Apply rigorous statistical methods to evaluation design, including sample sizing, confidence intervals, significance testing, and techniques for handling non-deterministic model outputs. Automated Grader Validation: Compare automated evaluation methods with human assessments to establish reliability, identify limitations, and improve scoring accuracy. Evaluation Infrastructure Collaboration: Partner with Engineering and Analytics Engineering to operationalize evaluation frameworks, integrate testing into development workflows, and support scalable evaluation infrastructure. Performance Analysis and Improvement: Interpret evaluation results, identify weaknesses and quality gaps, and provide actionable recommendations to improve model and agent performance. Quality Standards and Documentation: Establish evaluation guidelines, documentation practices, review processes, and consistent quality standards across AI development initiatives. Cross-Functional Partnership: Collaborate with Product, Engineering, Analytics Engineering, and business stakeholders to define success criteria and align evaluation priorities with business needs. Continuous Improvement and Mentorship: Promote evaluation best practices, share technical insights, and help foster a culture of measurable AI quality, accountability, and evidence-based decision-making. Safe AI Deployment: Support teams in making informed release decisions through independent, reliable assessments of model performance and readiness. Requirements: Approximately 6–10 years of experience in quantitative data science, machine learning, or a related technical discipline, with focused experience in measurement, evaluation, experimentation, or model validation. Strong quantitative and statistical foundations, including experience designing rigorous experiments and interpreting results with appropriate uncertainty measures. Demonstrated experience defining meaningful evaluation metrics, establishing ground truth, and assessing the quality of complex or ambiguous model outputs. Proficiency in Python and SQL, with experience applying these skills to data analysis, model assessment, and evaluation workflows. Experience evaluating machine learning models in production environments and translating findings into practical improvements. Ability to validate automated graders against human judgments and account for variability, non-determinism, and potential measurement bias. Strong problem-solving and analytical judgment, particularly when developing evaluation approaches for new or evolving AI systems. Excellent communication skills, with the ability to explain evaluation results, trade-offs, and recommendations to technical teams and business stakeholders. Proven ability to collaborate across functions while independently owning complex analytical projects in a fast-paced environment. A quantitative degree in data science, statistics, mathematics, computer science, or a related field is an advantage; equivalent practical experience is also welcome. Hands-on experience evaluating large language models (LLMs) or AI agents in production, including evaluation harnesses, LLM-as-judge calibration, and continuous integration regression gates, is a plus. Experience evaluating text-to-SQL systems, analytics agents, or other AI applications where outputs can be verified against underlying data is an advantage. Background in fintech, brokerage, financial services, or other domains where incorrect outputs can create significant business or risk consequences is desirable. Familiarity with AI tools used in research, analysis, and engineering workflows is beneficial. Benefits: Competitive compensation: Salary package designed to reflect experience and expertise. Stock options: Opportunity to participate in the company's long-term growth. Health benefits: Benefits designed to support employee health and well-being. Home-office setup allowance: One-time allowance of USD $500 to support your remote workspace. Monthly stipend: USD $150 per month provided through a Brex card. Remote work environment: Opportunity to work remotely within the Americas, with Canada as the target location for this listing. Technical ownership: Lead the development of evaluation practices and establish standards that influence AI quality across the organization. Cross-functional impact: Work closely with product, engineering, analytics, and business teams to turn rigorous measurement into better AI systems. Professional development: Build expertise in AI evaluation, model validation, and the responsible deployment of intelligent systems in a growing technology environment.