AI Assists in Formal Software Verification

The software industry has long struggled with the high cost of formal verification, a process that mathematically proves code operates without errors. Recent developments suggest that artificial intelligence may be altering that economic landscape. Anthropic has announced that it used its Claude model to generate mathematical proofs demonstrating that specific existing software behaves correctly. This achievement marks a significant step toward making AI generated software more trustworthy, although the company emphasizes clear limitations on what these proofs guarantee.

Historically, formal verification has been prohibitively expensive and time consuming. The technique, which involves proving that software runs without error in all circumstances, has been confined largely to extreme cases such as avionics systems for aircraft. For instance, specialists required twenty person years of work to verify seL4, a small operating system kernel. This high barrier to entry prevented the method from becoming a standard practice in general software development.

The current project represents a shift in how this work is conducted. Anthropic’s Frontier Red Team collaborated with a partner named Theorem to delegate the proof generation to Claude. The model produced approximately 180 million lines of mathematical proof code within a matter of weeks. This work relied on test programs from 123 open source libraries that are part of Google’s OSS-Fuzz security project. The speed of this process contrasts sharply with the decades long timelines previously associated with such tasks.

It is important to note that Claude did not write new software. Instead, it verified existing code by converting crash tests into mathematical problems. The model analyzed compiled machine code and constructed extensive proofs to support its findings. A separate verification program called Rocq automatically checks these proofs. Anthropic states that this process requires almost no human reviewers. While the tools and formal foundations were built by humans at Theorem, the researchers describe the proof generation itself as fully autonomous.

During the verification process, Claude identified hundreds of bugs and was permitted to fix them. Many of these errors would likely have escaped conventional testing because they only appear under narrow conditions. Such conditions include unusually large input files or specific sequences of data accesses. In one notable instance, the team verified a widely used PDF reader. They found as many security bugs in a single week as are usually reported for that software in an entire year. In another application, the model identified a logical error that had gone unnoticed for more than decade.

Anthropic maintains that even when a complete proof fails, the attempt can still reveal real bugs. This suggests that the value of the process extends beyond simple pass or fail results. The ability to uncover subtle, hidden defects highlights the potential of this approach for improving software reliability. The findings indicate that AI driven verification can detect issues that traditional testing methods often miss.

However, the company is careful to clarify that formally verified does not mean error free. The proofs hold only under specific, defined conditions. These include a particular processor architecture, single threaded programs, and inputs smaller than 64 MB. Real world deployments of these libraries often run on different hardware and configurations. Therefore, the verification results are context specific and do not guarantee universal correctness across all possible environments.

Cost remains a significant constraint for this technology. Individual proofs consume hundreds of millions of tokens, and Anthropic has not published the total project costs. Despite this, the company argues that if AI can verify its own code, developers could trust AI generated software more. This trust could lead to less time spent on manual reviews, cutting costs across the coding process. A smaller test run indicated that proofs can be reused, and efficiency is improving. Opus 5.5 solves half the proofs using about 40 percent fewer tokens than Opus 5.

The project shifts the focus from whether AI can program to whether it can prove the code it writes is safe. Anthropic expects formal verification with AI to become as practical as standard testing in the future. Theorem’s roadmap already targets critical infrastructure components, including glibc, OpenSSL, and cURL. As these technologies mature, the integration of AI into the verification pipeline may become a standard feature in software development.