An open model beat Claude Code at finding access-control bugs

Semgrep ran a benchmark on one narrow but common security task: finding IDOR bugs, the access-control flaws where a user can read or change records that should not be theirs. The result that drew attention is that GLM-5.2, an open-weight model from Zhipu, scored 39 percent on their F1 measure, ahead of Claude Code's best run at 37 percent (on Opus 4.6) and 28 percent on Opus 4.8 and 4.7. It did this for about $0.17 per vulnerability found, with no special scaffolding: just the system prompt and access to the codebase.

The headline number is not the real lesson. Semgrep's own pipeline, which wraps a frontier model in a custom harness that discovers endpoints and feeds the model structured context, scored 61 percent on GPT-5.5 and 53 percent on Opus 4.8. The biggest gaps in the whole test were between differently built systems, not between models. As the team puts it, the harness still matters more than the model. The open-weight models were tested without the endpoint-discovery scaffolding the proprietary pipeline uses, so part of the comparison is about plumbing rather than raw model skill.

Two findings are worth holding together. A cheap open model can now match or beat a frontier coding agent on a real security task when both run bare, and a good harness on top of either still beats both by a wide margin. Semgrep's writeup has the full table.

Why it matters

If you run security scanning, the model you pick matters less than the system you build around it, so spend your effort on endpoint discovery and context, not just on buying the most expensive model. An open model at $0.17 per finding is now worth testing on your own code.