CWE-bench v2: More coverage, quality, and transparency

Why it matters

Collinear AI has released CWE-bench v2, a public benchmark of security tasks in which an agent receives a real open-source repository and must find and fix a weakness without breaking existing functionality. The set spans a broad range of vulnerability classes across many distinct weakness types, and scoring is fully programmatic, making results deterministic and reproducible. A held-out set lets the team track progress on unseen tasks. Every task was manually reviewed against quality criteria, including whether instructions read realistically, whether verifiers check behavior rather than a specific fix structure, and whether memorized upstream patches fail. The grader and harness are open source, and the tasks can run directly from the Hugging Face Hub via Harbor. The environments are also tagged for reinforcement learning of defensive agents. The results show most tasks are solvable by at least one model, yet the leading model still fails a meaningful share on every attempt, indicating a long tail of unsolved cases. This matters because it offers a transparent, community-verifiable way to measure defensive cyber capabilities.