Content Specialist III | AI Evaluation & Prompting

Accountabilities: Test new AI model versions across a broad range of topics, use cases, and conversation scenarios. Evaluate AI-generated responses against established rubrics, guidelines, and quality standards. Identify, document, and communicate examples of both successful and unsuccessful model behavior. Write, test, and refine system prompts designed to influence model personality, tone, responses, and behavior. Assess whether evaluation rubrics and quality frameworks accurately measure model performance and identify opportunities for improvement. Conduct quality reviews of human- and agent-based evaluations to ensure accuracy, consistency, and compliance. Perform hands-on experiments with AI models and products to investigate capabilities, limitations, and unexpected behaviors. Investigate model failures, identify potential causes or patterns, and document findings clearly. Apply detailed instructions and evaluation criteria consistently across a high volume of work. Perform fact checking and verify model claims against reliable original sources when required. Translate observations and research findings into clear, actionable feedback for product and AI teams. Adapt evaluation approaches as model behavior, product priorities, and quality requirements evolve. Requirements 5+ years of professional experience in writing, editing, journalism, production, linguistics, STEM, coding, policy, or another relevant subject-matter discipline. Bachelor’s degree or equivalent professional experience. At least 1 year of hands-on AI experience is preferred, including prompting, annotation, model evaluation, red teaming, or related work. Strong experience evaluating AI-generated content, large language model outputs, or other complex digital content is highly desirable. Deep expertise in at least one subject area, with the ability to evaluate content accurately as a subject-matter expert. Strong writing, editing, fact-checking, and research capabilities. Experience writing and refining prompts and assessing how prompt changes influence AI behavior. Strong analytical judgment and exceptional attention to detail. Ability to consistently apply detailed rubrics, guidelines, and evaluation criteria across high-volume work. Ability to identify subtle differences in AI responses and determine why a response succeeds or fails. Strong documentation and communication skills, with the ability to clearly explain findings, questions, blockers, and recommendations. Ability to verify claims against original or authoritative sources. Experience conducting detailed failure investigations is preferred. Comfortable working independently while escalating important questions, risks, and blockers early. Highly adaptable and comfortable shifting priorities as AI products and business needs evolve. Professional fluency in English is required. Must be authorized to work in the United States without requiring current or future sponsorship. Benefits 6-month remote contract based in the United States. Full-time schedule of approximately 40 hours per week . Standard schedule of Monday through Friday with 8-hour shifts. Health insurance. Dental insurance. Vision insurance. Health savings account (HSA). Life insurance. Paid time off. Retirement plan. Opportunity to work directly on the evaluation and improvement of advanced AI systems. Hands-on exposure to AI model evaluation, prompt engineering, quality frameworks, and emerging AI products. Opportunity to apply specialized subject-matter expertise to real-world AI development and product improvement.