Anthropic's Claude Code vs. OpenAI's Codex: AI Coding Agents Lack Time Sense and Self-Assessment, MATS Research Finds

AI Coding Agents Struggle with Time and Self-Correction

A new study comparing Anthropic's Claude Code and OpenAI's Codex reveals that AI coding agents lack a sense of time and cannot accurately judge their own work, consistently overestimating task runtimes and overrating output quality. This research, published via LessWrong and the MATS (Machine Alignment & Theory Scholars) program, highlights critical operational consequences for long-running, semi-autonomous coding jobs.

Methodology: Testing Leading AI Models

The study rigorously tested two prominent AI coding agents: Anthropic's Claude Code and OpenAI's Codex. Researchers subjected these models to 200 distinct tasks drawn from ProgramBench, a standard benchmark suite, alongside an additional 18-benchmark suite. This comprehensive testing aimed to evaluate the agents' performance across a variety of coding challenges and to observe their operational characteristics, particularly concerning time management and self-assessment.

Inaccurate Time Estimation

One of the most striking findings was the agents' consistent inability to accurately estimate task runtime. Claude Code, for instance, overestimated task duration by approximately three times the actual completion time. OpenAI's Codex exhibited even greater discrepancies, overestimating by a factor of six to ten times. The study noted that these time estimation errors were most pronounced on shorter tasks, suggesting a fundamental challenge in gauging the scope and duration of even minor coding assignments. This lack of a reliable internal clock has direct operational consequences, especially for long-running or time-sensitive development projects.

Self-Assessment Flaws and Overconfidence

Beyond time estimation, the research also highlighted a significant flaw in the agents' ability to judge their own work. On average, AI coding agents graded their own output about 20 points higher than their actual scores. In some cases, agents rated tasks that had clearly failed with scores approaching 70%, despite actual performance being under 15%. This overestimation of quality, even for demonstrably incorrect solutions, poses a challenge for integrating these tools into workflows where independent verification is not consistently applied.

Impact of the Software Harness

The study also observed that the operational behavior of the AI models was significantly influenced by their "harness" – the software environment in which they operated. For example, Claude Code typically worked for a median duration of approximately 90 minutes before stopping, whereas Codex ceased operations after roughly 30 minutes. This variation suggests that the surrounding infrastructure and operational parameters play a crucial role in how these AI coding agents perform and manage their tasks, rather than the core model alone dictating behavior.

Comparison of Claude Code and Codex Performance

Feature	Anthropic's Claude Code	OpenAI's Codex
Time Overestimation	~3 times	6-10 times
Median Runtime	~90 minutes	~30 minutes
Self-Assessment Accuracy	Overestimated by ~20 points	Overestimated by ~20 points

Key Takeaways for Developers and Organizations

• AI coding agents currently lack an intrinsic sense of time, leading to significant overestimations of task duration.
• These agents struggle with accurate self-assessment, often overrating the quality of their own code, even for failed tasks.
• The operational behavior and runtime of AI models can vary substantially based on their software environment or "harness."
• Organizations deploying AI tools for code assistance should implement robust external validation and monitoring.
• Reliance on AI agents for fully autonomous, long-running tasks without human oversight may lead to inefficiencies and errors.

Conclusion

The findings from this study underscore that while AI coding agents like Claude Code and Codex offer considerable potential, they are not yet capable of fully autonomous, self-regulated work, particularly concerning time management and quality assurance. Developers and organizations integrating these AI tools into their workflows must account for these limitations by implementing external monitoring, validation, and human oversight. As AI technology continues to evolve, addressing these fundamental challenges will be crucial for enhancing the reliability and practical utility of AI in software development.

Sources

• Professional Software Developers Don’t Vibe, They Control: AI Agent Use for Coding in 2025
• How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests
• The new AI credit/token model is a serious step backwards for AI-assisted coding · community · Discussion #197872 · GitHub
• fmind/ai-coding-5xp-template
• Coders are refusing to work without AI — and that could...