AI proved your code correct. What did it actually prove?

Why it matters

Keller AI, a publication focused on checking software against requirements and policies, uses recent projects from Anthropic and GitHub as a jumping-off point to explain formal verification to a general engineering audience. Anthropic's collaboration with Theorem converted existing fuzz harnesses into formal statements about compiled programs, with Claude constructing proofs that a proof assistant then checked. GitHub Next and Microsoft Research are exploring a related workflow in which a human reviews a specification and an agent's implementation is checked against it. The piece walks through a sorting example to show why passing tests on sample inputs does not establish universal claims, and how induction can prove properties for every finite input. It then uses Euclid's first proposition to show how proofs build from definitions, assumptions, and previously established results, and how modern proof assistants check each step. The core lesson is that verification guarantees depend entirely on the specification chosen and the assumptions stated, so readers must interpret what was actually proved and what remains unverified.