AI Response Quality Evaluator

Accountabilities: Evaluate AI-generated responses using personalized prompts and information retrieved from connected Google applications. Assess whether responses are relevant, accurate, helpful, contextually appropriate, and sufficiently personalized. Identify incorrect personalization, unsupported assumptions, irrelevant recommendations, inconsistencies, and other quality issues. Compare multiple AI-generated responses and assess differences in quality and overall user experience. Review nuanced AI outputs to identify both strengths and weaknesses, including subtle issues that may not be immediately apparent. Provide clear, detailed, structured, and well-reasoned feedback to support improvements to AI models and personalization systems. Apply project-specific evaluation guidelines consistently to ensure reliable and high-quality assessments. Maintain confidentiality and follow all applicable data privacy and security requirements throughout the project. Work independently and manage assigned evaluation tasks effectively within the project timeframe. Requirements Currently based in Australia. Willing and able to connect relevant Google applications to Gemini, subject to required consent and permissions. Active use of Google applications such as Gmail, Google Calendar, Google Photos, and Google Drive. Sufficient personal data or usage history within connected Google applications to support personalized retrieval evaluations. Strong analytical and critical-thinking skills, with excellent attention to detail. Excellent written English and the ability to communicate observations and evaluation results clearly. Ability to distinguish factual accuracy, relevance, personalization quality, and subtle contextual errors. Strong ability to follow detailed guidelines and apply evaluation criteria consistently. Comfortable working independently in a fully remote environment. Access to a desktop or laptop with a reliable internet connection. Bachelor’s degree or equivalent practical experience in any field. Ability to handle potentially sensitive personal information responsibly and maintain strict confidentiality. Availability to work with at least 4 hours of overlap with Pacific Standard Time (PST). Benefits Fully remote contractor position available to candidates based in Australia. Project duration of up to 16 weeks. Flexible remote working environment suited to independent work. Opportunity to contribute directly to the evaluation and improvement of AI personalization capabilities. Exposure to advanced AI evaluation workflows involving personalized prompts and contextual information. Opportunity to develop practical experience assessing AI-generated content and user experiences. Structured project guidelines and evaluation criteria to support consistent assessments. Contractor engagement with a defined project scope and onboarding process. Shortlisted candidates receive a Job Interest Form before selection and onboarding. Selected candidates receive further information regarding consent and onboarding requirements.