New AI programming benchmark tests code mergeability; 40% of attempts with Opus 5.5 still failed to pass.

Beating AI Express (Dongcha) reports: Artificial Analysis has launched the FrontierCode 1.1 programming benchmark, which primarily assesses whether AI-generated code meets the merge standards of open-source project maintainers. Claude Opus 5.5 took first place with a 52.2% overall score, though only 58% of its attempts passed all mandatory checks—meaning over 40% failed to meet requirements. The benchmark was developed by Cognition in partnership with maintainers from 36 open-source projects, with each task refined for more than 40 hours. It goes beyond checking if code runs: it requires new tests to identify issues in existing code, forbids modifying unrelated files, and ensures no disruption to original functionality. Artificial Analysis selected 100 of the most challenging private tasks, had all models use the same Agent framework, and ran each task three times. No tested model successfully completed 9 of these tasks. Cost gaps are even wider than score differences: GPT-6.1 Sol scored 49.1%, just 3.1 percentage points lower than Opus 5.5, but its average per-task cost was only $0.81, compared to $10.15 for Opus 5.5—an over 12x difference. Xiaomi’s MiMo-V2.6-Pro led open-source models with a 45% score and a per-task cost of just $0.18; DeepSeek V4.1 Flash and Kimi K3 both scored 44%. A key caveat: The 52.2% overall score is not the actual pull request (PR) merge rate. The benchmark uses automated checks and reviews by the GPT-6 Luna model, not line-by-line reviews by human maintainers. All models used a unified Agent framework, so this ranking cannot directly reflect the real-world performance of each vendor’s official programming tools.