Senior AI Engineer

The Senior AI Engineer will own evaluation and governance capabilities for a core platform where AI agents take real actions, including sending messages, updating records, and preparing decisions. You will ensure each agent capability is evaluated before release and after model or instruction changes, with clear thresholds and go-live controls. You will design and implement runtime safeguards that enforce accountability, approvals, data restrictions, deterministic checks, and quality-based suspension in code. The role requires strong production experience with tool-using, multi-step LLM agents, applied statistics, and maintainable software services. You will work closely with platform engineering, cloud engineering, product owners, and client engagement teams to convert governance rules and acceptance scenarios into executable validation. Your work will help establish reliable, auditable, and provider-independent AI systems. Key Responsibilities: - Build an evaluation framework with datasets, metrics, agreed thresholds, and a go-live gate for each agent capability. - Re-evaluate agent capabilities after every model or instruction change and identify quality regressions. - Build a governed agent runtime with accountable ownership for each action, human approval where required, and deterministic checks before irreversible actions. - Implement rule-level or campaign-level approval for high-volume actions, including statistically sized review samples before release. - Configure automatic suspension when quality thresholds are not met. - Design review and evaluation flows for AI-generated content sent externally. - Record every model call, including the model, instruction version, data provenance, and quality checks. - Maintain platform independence from any single model provider. - Convert written governance rules, ontologies, and acceptance scenarios into executable validation with the Principal Platform Engineer. - Collaborate with cloud engineering on runtime infrastructure, isolation, and model-provider connectivity, and with product stakeholders on capability requirements, datasets, and thresholds. Requirements: - Senior-level experience building and operating production AI systems, including tool-using, multi-step LLM agents. - Production experience designing evaluation sets, metrics, quality thresholds, go-live gates, and regression detection across model or prompt changes. - Strong applied statistics skills, including review sample sizing, confidence bounds, and error-rate estimation from samples. - Experience designing safety controls enforced in code, including approvals, deterministic checks, and kill switches. - Strong Python or TypeScript skills, including testing, code review, and development of maintainable services. - Ability to write clear design notes and document technical decisions before implementation. - Hands-on experience with Agent Orchestration, LLM Evaluation, Evaluation Metrics, Software Testing, Safety Engineering, Runtime Governance, Approval Workflows, Model Observability, Data Provenance, Model Provider Abstraction.