Semgrep: GLM 5.2 beats Claude in our Cyber Benchmarks
Semgrep conducted benchmarks comparing open-weight AI models against proprietary ones like Claude for vulnerability detection. The results showed that Zhipu AI's GLM 5.2 model performed surprisingly well, suggesting that model architecture is becoming increasingly competitive with frontier models.
Why it matters
This demonstrates the rapid closing gap between open-weight models and expensive proprietary AI in specialized security tasks.
We ran a set of popular open-source models against our IDOR benchmark, the same dataset and the same prompt we've used to evaluate frontier coding agents. The result surprised us: GLM 5.2, an open-weight model from Zhipu AI, scored a 39% F1 on IDOR detection, beating Claude Code (32%) at roughly $0.17 per vulnerability found. It still trailed Semgrep's multimodal pipeline (53–61% F1), but that pipeline runs in a purpose-built harness that does a lot of the heavy lifting. Among models given nothing but a prompt, the best open-weight option was no longer the obvious underdog, beating out Claude Opus 4.8.
The article presents technical benchmark data and methodology without taking a political stance.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in