Artificial Analysis Intelligence Index v5 Overhaul: Private Test Sets Widen Open vs. Closed AI Model Gap to 12 Points

Artificial Analysis overhauled its Intelligence Index to version 5 on October 28, 2026, incorporating 60% private test sets to prevent "benchmaxxing" and revealing a 12-point performance gap between open and closed AI models, up from 3 points. For broader context, explore our Top 100 AI Tools.

Methodology Shift: Private Test Sets and Their Impact

The core change in the Intelligence Index v5 is the integration of 60 percent private test sets. This strategic move by Artificial Analysis addresses concerns that public benchmarks can inadvertently lead to models being over-optimized for those specific tests, rather than demonstrating broader intelligence. By making a majority of the evaluation criteria private, Artificial Analysis seeks to provide a more robust and less exploitable assessment of AI model performance.

However, this shift also introduces a trade-off. A downside of using private test sets is that external auditors cannot independently review the prompts for potential errors. Artificial Analysis itself had to revise flawed questions within its AA-Omniscience knowledge test, highlighting the importance of internal rigor in such a system.

Performance Revisions: Open vs. Closed Model Gap Expands

The most striking outcome of the Intelligence Index v5 update is the expansion of the performance gap between open and closed AI models. Previously, this gap was measured at 3 points; with the new methodology, it has increased to 12 points. This revision suggests that prior public benchmarks may have presented open models as being closer to the cutting edge than their actual capabilities indicated.

AI researcher Nathan Lambert has observed that many training advancements in AI models often stem from reinforcement learning environments that closely resemble existing benchmarks. This observation underscores the rationale behind Artificial Analysis's move to private test sets, aiming to mitigate the influence of such benchmark-specific optimizations.

Notable Model Performance Changes

The updated Intelligence Index v5 has led to significant shifts in the rankings of several prominent AI models:

• Kimi K3.1 Pro: This model experienced a drop of five spots, moving from fourth to ninth place. It now ranks behind eight closed models, indicating a substantial re-evaluation of its performance relative to its peers. For more information on Kimi, you can visit its tool page.
• Grok 4.7: This model saw a loss of 12 points in the new ranking, reflecting a notable decrease in its assessed performance under the revised evaluation criteria.

These changes highlight how a shift in benchmarking methodology can significantly alter the perceived standing of AI models, particularly when moving away from publicly accessible test environments.

Implications for AI Development and Evaluation

The overhaul of the Artificial Analysis Intelligence Index v5 underscores an ongoing challenge in AI evaluation: creating benchmarks that accurately measure general intelligence rather than specific test performance. The move to private test sets represents an effort to push models towards more generalized capabilities, reducing the incentive for developers to simply optimize for known public datasets.

While the increased gap between open and closed models may seem concerning for the open-source community, it also provides a clearer picture of the current state of AI development. This transparency, even if it reveals a wider disparity, can inform future research and development efforts, guiding them towards areas that require more fundamental advancements rather than benchmark-specific tuning.

Conclusion

The Artificial Analysis Intelligence Index v5, released on October 28, 2026, marks a pivotal change in how AI models are evaluated. By incorporating 60 percent private test sets, Artificial Analysis aims to provide a more accurate and less exploitable measure of AI intelligence. This methodological shift has resulted in a widened performance gap between open and closed models, now at 12 points, and has led to significant ranking adjustments for models like Kimi K3.1 Pro and Grok 4.7. The ongoing evolution of benchmarking methodologies remains crucial for fostering genuine progress in artificial intelligence.

Sources

• scicode-verified: how benchmark defects un
• ainews-web-2025/src/content/issues/26-07-08-grok-45.md at main · smol-ai/ainews-web-2025 · GitHub
• Inference Economics of Enterprise Coding Agents: A Case Study of Cloud vs. On-Premise LLMs
• Introducing Gemini 4 Carbon