Meta's Chief AI Officer Alexandr Wang refutes SemiAnalysis' 'benchmark-rigging claim', stating that old tests are already saturated.

Beating AI Insight News: AI research firm SemiAnalysis has singled out Gemini 3.8 Flash and Muse Spark 1.3 as among the most obvious "benchmaxxed" models—meaning they are over-optimized for public benchmarks. The two models scored 89.4% and 88.8% respectively on Terminal-Bench 2.1, figures close to those of GPT-6 Astra and Claude Fable 5.1. However, their scores plummeted to just 19.1% and 33.3% on the newly launched Terminal-Bench 4.0, while Astra and Fable 5.1 retained scores of 57.7% and 55.8%. SemiAnalysis suspects the problem stems from public benchmarks becoming increasingly vulnerable to targeted optimization. All tasks in Terminal-Bench 2.1 are public; even if model developers do not directly use test questions for training, they can purchase reinforcement learning environment data highly similar to these tasks, leading models to excel at this "test" but fail to generalize to new tasks. That said, SemiAnalysis has not produced direct evidence that Google or Meta bought such data. Meta’s Chief AI Officer Alexandr Wang pushed back, calling the argument "stupid". He cited that GPT-5.6 Sol also dropped from 88.8% on Terminal-Bench 2.1 to 37.3% on 4.0, explaining that 2.1 is saturated and can no longer effectively differentiate top-tier models, while 4.0 is far from saturated. Terminal-Bench 4.0 removed 8 tasks that were saturated, had existing public solutions, or other issues, modified 19 questions, and adjusted resource limits including time, CPU, and memory.