SWE-Bench AI Task Auditor - Freelance AI Trainer Project

Accountabilities: Evaluate software engineering tasks for technical accuracy, realism, solvability, reproducibility, and alignment with SWE-Bench standards. Review task codebases, integrations, tests, and evaluation criteria to identify potential technical weaknesses. Rigorously test and troubleshoot complex technical scenarios to determine whether tasks function as intended. Investigate codebase integration problems, test failures, logic errors, and other implementation issues. Provide clear, precise, and actionable feedback that enables task creators to correct identified problems. Apply professional software engineering judgment to assess whether tasks reflect realistic development scenarios. Help maintain a high standard of technical rigor and accuracy across AI training and evaluation workflows. Requirements Demonstrable professional experience in software engineering, including experience navigating complex codebases and developing real-world applications. Strong knowledge of software engineering principles and familiarity with SWE-Bench-style tasks and evaluation workflows. Strong analytical and problem-solving abilities, with the capacity to investigate complex technical scenarios systematically. Experience troubleshooting code, diagnosing test failures, and identifying underlying logic or integration issues. Ability to evaluate technical work objectively and communicate findings clearly and constructively. Strong attention to detail and a rigorous approach to technical validation. Ability to work independently and manage project-based assignments effectively. Deep expertise in one relevant software engineering specialty is sufficient; expertise across multiple domains is not required. A secure computer and reliable, high-speed internet connection suitable for remote technical work. Benefits Fully remote freelance contract. Flexible project-based working environment. Opportunity to contribute directly to the training and evaluation of AI systems. Ability to apply real-world software engineering expertise to challenging technical tasks. Compensation of $60/hour , with the final rate determined based on experience, expertise, and geographic location. No company-sponsored health insurance, PTO, or other employee benefits, as this is a freelance contractor position.