It is still far from truly replacing programmers: AI agents accessing enterprise private code only achieve a maximum pass rate of 40%.

Insight Beating AI News Flash: Y Combinator (YC)-incubated AI data firm Specific Labs has launched Real-SWE, a coding evaluation benchmark that tests coding agents on real private code from actual enterprises. The tasks are sourced directly from production environments, including tax calculation, bill migration, API billing, and customer data migration. Neither the code nor answers are publicly available, so agents must independently master the company’s business rules and code structure. Results show Fable 5.1 ranked first with a 38.8% pass rate; GPT-6 Astra followed at 33.8%, Gemini 3.8 Flash at 31.2%. GLM 5.3 took fourth place with 28.8%, outperforming Grok 4.6, Kimi K3, and GPT-5.6 Sol. Of the 10 currently public tasks, 6 have an overall pass rate below 15%, and one task saw all models fail. The standout performer was GLM 5.3. Real-SWE assesses agents’ ability to continuously read code, identify rules, and complete multi-step modifications in unfamiliar projects. Zhipu AI researcher Xiaopu Peng noted that his team has been training long-range tasks since GLM-5.1, and Real-SWE is more aligned with such capabilities than public SWE benchmarks, which partly explains GLM 5.3’s strong performance here.