OpenAI Audit Reveals ~30% of SWE-Bench Pro Tasks Are Broken, Undermining AI Model Evaluations

OpenAI's audit of SWE-Bench Pro, a key benchmark for evaluating AI coding agents, revealed that approximately 30% of its tasks are broken, undermining the reliability of AI model evaluations. This finding, detailed in a report published on July 8, 2026, follows OpenAI's earlier identification of fundamental design and contamination issues with SWE-Bench Verified. For broader context, explore our AI News.

The Scope of the Problem: Broken Tasks Identified

The audit employed a two-pronged approach to identify issues within SWE-Bench Pro. Initially, an automated data-quality pipeline flagged 200 out of 731 tasks, representing 27.4% of the benchmark, as broken. Following this, a more intensive human annotation campaign involved five experienced software engineers per task. This human review identified an even higher number of issues, pinpointing 249 broken tasks, or 34.1% of the total.

The identified problems were diverse, ranging from overly strict tests that incorrectly failed valid solutions to underspecified prompts that lacked sufficient detail for AI agents to complete tasks accurately. Other issues included low-coverage tests that did not adequately validate solutions and misleading prompts that directed AI models toward incorrect approaches.

Implications for AI Model Evaluation and Safety

The discovery of widespread flaws in SWE-Bench Pro has significant implications for how the AI industry assesses the capabilities of AI coding tools. Accurate benchmarks are considered crucial for making informed decisions regarding AI safety and deployment, particularly under frameworks like OpenAI's Preparedness Framework. When evaluation tasks are compromised, assessments of model capabilities become unreliable, potentially leading to an overestimation or underestimation of an AI's true performance.

This is not the first time OpenAI has highlighted issues with coding benchmarks. The organization previously identified fundamental design and contamination problems with SWE-Bench Verified, another benchmark. These recurring issues underscore a broader challenge in developing robust and trustworthy evaluation methods for advanced AI systems.

Challenges in Measuring AI Progress

The integrity of benchmarks directly impacts how the AI industry measures its progress. If the tools used to evaluate AI models are themselves flawed, then the reported advancements and comparative performance metrics may not accurately reflect reality. This situation can hinder genuine innovation and make it difficult to identify truly superior AI solutions.

OpenAI advises model developers to exercise caution and carefully examine their results rather than relying solely on benchmark scores at face value. This recommendation emphasizes the need for a more critical and nuanced approach to AI evaluation, moving beyond simple numerical rankings.

Conclusion

OpenAI's audit of SWE-Bench Pro serves as a critical reminder of the challenges inherent in developing reliable benchmarks for AI. The finding that approximately 30% of tasks are broken highlights the need for continuous scrutiny and improvement in evaluation methodologies. As AI capabilities advance, the accuracy and robustness of benchmarks will remain paramount for ensuring responsible development and deployment.

Sources

• Do Androids Dream of Breaking the Game? Systematically...
• Dissecting the SWE-Bench Leaderboards