Veriwire
← Back to the wire

Why Your AI Agent Can't Solve CTF Challenges: Pwnable — TeamH4C

SpeculationResearchNov 7, 2026

TeamH4C reports that with the model fixed at Kimi-K2.6, Pwnagent captured a shell or flag in 62.12% of 66 local pwn CTF challenges with dynamic measurement and diagnosis-guided repair, versus 42.42% without both. The analysis shows solving performance shifts with the harness when models are fixed and with models when harnesses are fixed, and examines where information gathering, exploit design, execution, and validation fail between vulnerability discovery and working exploits.

Why it matters

TeamH4C's article examines why AI agents often fail at pwn CTF challenges even after finding vulnerabilities. The piece reviews Pwnagent, an automatic exploit generation system that assigns analysis, dynamic measurement, and code generation to specialist agents, and compares configurations with and without dynamic measurement and diagnosis-guided iterative repair while keeping the underlying model fixed. It also draws on the EnIGMA agent study of tool interfaces and the ExploitGym benchmark comparing model changes under a fixed harness. The central finding is that solving performance depends on the combination of model and harness: results shift when the harness changes with the model fixed, and when the model changes with the harness fixed. The article then traces failure points between vulnerability discovery and a working exploit, including missing analysis inputs, misread execution states, and weak failure diagnosis. It is the third installment in a series on why AI agents struggle with CTF challenges.

Receipt № 27851 source · awaiting confirmation ◐

Evidence

1source· awaiting independent confirmation

No score is assigned. Sources and their independence are shown in the citation chain below.

Citation chain · 1 source

01med
h4c.teamBloghttps://h4c.team/en/posts/97This source provides TeamH4C's detailed breakdown of failure points in AI-driven pwn exploit generation, emphasizing that agent performance reflects the model-and-harness combination rather than model capability alone.
PRIMARY
TeamH4CCompany

Related

Nov 7Why a new report on Anthropic automating AI development is alarming researchersNov 7Agent memory needs its OpenTelemetry moment — LoreKit blogNov 8LLMs are terrible at designing AI systems
Canonical: https://h4c.team/en/posts/97