Attorney, Evaluation Operations (Contract)

Accountabilities: AI Output Evaluation: Compare multiple AI-generated responses to identical legal tasks, including legal memoranda, contract redlines, and research answers, assessing their accuracy, relevance, completeness, and overall quality against established evaluation rubrics. Product Testing and Feedback: Test new legal technology features and workflows in pre-release environments, providing detailed qualitative feedback based on professional legal experience. Evaluation Rubric Review: Review and assess grading rubrics to ensure they are accurate, comprehensive, consistent, and appropriate for evaluating legal AI outputs. Dataset Development: Create and contribute to evaluation datasets containing realistic legal queries, contract markups, and synthetic legal documents to support the testing and improvement of AI systems. Legal Research and Analysis: Apply practical legal knowledge and analytical judgment to assess AI-generated legal content and identify potential inaccuracies, gaps, or areas requiring improvement. Cross-Functional Collaboration: Work closely with legal product specialists and evaluation teams to support research initiatives and provide insights that inform product development and key technical decisions. Project-Based Delivery: Complete assigned evaluation projects within agreed deadlines, often involving short turnaround times of 24–48 hours. Projects generally require between 2 and 20 hours over several days, with the flexibility to accept or decline individual assignments. Requirements Juris Doctor (JD) degree and an active license to practice law in at least one U.S. jurisdiction. At least 3 years of professional legal experience at an AmLaw 100 or comparable law firm, or within an in-house legal department. Professional experience in either commercial litigation or transactional law. Familiarity with artificial intelligence tools, including practical experience using AI applications in a professional or personal context. Excellent legal writing skills, with the ability to communicate complex legal concepts and reasoning clearly, concisely, and accurately. Strong analytical and critical-thinking abilities, particularly when evaluating the quality, reliability, and completeness of legal information. Excellent attention to detail and the ability to apply structured evaluation criteria consistently. Ability to work independently, manage project deadlines, and deliver high-quality feedback in a remote environment. Availability to accept occasional short-notice assignments and work on weekends when project deadlines require it. Flexibility to manage a variable workload, as projects are assigned on an as-needed basis without a minimum hours commitment. Preferred Qualifications: Previous experience using Harvey or other AI-powered legal research and productivity tools in professional practice. Experience evaluating, grading, annotating, or reviewing AI-generated content. For litigation attorneys: proficiency with LexisNexis or Westlaw and experience conducting legal research and cite-checking. A combination of law firm and in-house legal experience. Benefits Competitive hourly compensation: $125 USD per hour, based on hours worked and invoiced. Flexible, project-based workload with no minimum hours commitment. Fully remote working arrangement within the United States. Freedom to manage your own schedule within individual project deadlines. Ability to accept or decline projects based on your availability. Access to pre-release legal technology environments and innovative AI-powered tools. A dedicated laptop provided for project work. Six-month independent contractor engagement (1099). Opportunity to contribute to the development and evaluation of emerging legal AI technologies. Engagement Details: Contract type: Independent contractor (1099), engaged directly by the partner company. Duration: 6 months. Working hours: Part-time and project-based, with no guaranteed minimum. Payment: Contractors invoice for completed hours. Work arrangement: Fully remote, United States-based. Benefits: This independent contractor position does not include employee benefits. Scope limitations: No client representation, court appearances, litigation work, or business development responsibilities.