Principal Applied Scientist

Accountabilities: Design and lead an end-to-end evaluation framework for planner-executor agents, covering scenario definition, user simulation, configuration, execution, grading, comparison, and insight generation. Define scenario schemas, grading specifications, and validation processes in collaboration with engineering teams, ensuring scenarios capture goals, personas, starting states, and expected outcomes. Develop an LLM-powered user simulator capable of realistic multi-turn conversations, using hidden goals and stop conditions, and validate its behavior against production interaction logs. Build isolated, parallel experiment execution systems with configurable runs and adapters that route model traffic through existing LLM gateways. Develop and implement an initial selection of optimization plug-ins, such as failure-based model cascading, cache-aware model switching, and prompt caching. Instrument model calls and integrate tracing systems to attribute token consumption, cache utilization, latency, and costs to individual requests. Establish grading methods based primarily on verified end states, with optional LLM-based evaluation of answer and planning quality. Calculate and report key performance indicators, including verified success rates, cost per successful task, p95 latency, and failures by pipeline stage. Apply rigorous statistical methods, including paired comparisons, confidence intervals, win/loss/tie analysis, and non-inferiority testing, while clearly identifying inconclusive findings. Translate experimental results into projected financial impact at production scale and develop cost-versus-quality analyses to guide optimization decisions. Build an automated insight engine that uses statistical computation for all reported metrics and LLM-generated explanations to communicate evidence-backed findings. Develop a locally hosted, API-first dashboard with launch, live monitoring, results, comparison, insights, and tracing capabilities, supported by straightforward deployment and integration documentation. Lead staged experimentation campaigns covering production baselines, model screening, optimization testing, and validation of the strongest combinations. Assess how well agent performance generalizes across scenario categories, potentially incorporating data analytics benchmark scenarios. Deliver final recommendations with confidence intervals and projected dollar impact, alongside technical reports, runbooks, and integration guides. Collaborate with engineers and business leaders to establish evaluation criteria, success thresholds, and acceptable quality margins, communicating findings to both technical and non-technical stakeholders. Set experimental standards, provide technical leadership, and mentor engineers in rigorous agent evaluation and AI-assisted development practices. Requirements: Demonstrated experience with LLM and AI agent evaluation methodologies, including state-based verification, LLM-as-judge or agent-as-judge approaches, trajectory-level scoring, and user simulation. Hands-on experience building or evaluating production-grade planner-executor systems or other multi-step, tool-using AI agent architectures. Strong expertise in experimental design and statistical inference, including paired testing, non-inferiority analysis, power analysis, confidence intervals, and the ability to recognize when evidence is inconclusive. Experience with multi-objective optimization or cost-aware machine learning, including Pareto frontier analysis and translating technical results into projected financial savings. Familiarity with LLM routing, model cascades, model selection, and inference optimization techniques such as prompt and plan caching, context compaction, reasoning budgets, and cache-aware model switching. Understanding of LLM inference internals and the ability to evaluate the impact of optimization strategies on quality, latency, and cost. Experience implementing observability and tracing for LLM systems, including request-level attribution of tokens, cache usage, latency, and cost. Ability to develop production-quality research tooling using Python services, containerized and sandboxed execution environments, and data pipelines. Previous customer-facing or consulting experience, including scoping and delivering research-oriented technical engagements. Excellent communication skills, with the ability to explain complex technical findings, respond confidently to detailed questions, and connect results to business outcomes. Additional advantages include: Publications or open-source contributions in LLM evaluation, AI agents, or efficient inference. Experience developing internal dashboards or web applications that can be adopted and maintained by client engineering teams. Practical experience with LLM gateways and tracing platforms such as LangFuse or LangSmith. Background working in regulated or high-stakes enterprise environments where evaluation rigor is essential. A demonstrated understanding of real-world cost-quality trade-offs gained through production experience. Benefits: Annual salary range of $200,000–$300,000. Remote work opportunity within the United States. Customer-facing, research-driven work with substantial ownership of technical strategy, experimentation, and implementation. Opportunity to shape enterprise AI evaluation practices and deliver tools that support real-world deployment decisions. Close collaboration with engineering teams and business stakeholders. Significant technical leadership and mentoring responsibilities. Opportunity to work on advanced AI agent architectures, inference optimization, and measurable enterprise AI performance improvements. The role involves approximately 20% travel within the United States, including onsite collaboration and client-related activities. Willingness to attend an onsite final interview and participate in onsite onboarding is expected.