{"id":5737,"date":"2026-06-26T06:55:11","date_gmt":"2026-06-26T06:55:11","guid":{"rendered":"https:\/\/www.aviator.co\/blog\/?p=5737"},"modified":"2026-06-26T06:56:22","modified_gmt":"2026-06-26T06:56:22","slug":"verification-is-not-testing-understanding-the-difference-could-save-your-codebase","status":"publish","type":"post","link":"https:\/\/www.aviator.co\/blog\/verification-is-not-testing-understanding-the-difference-could-save-your-codebase\/","title":{"rendered":"Verification Is Not Testing. Understanding the Difference Could Save Your Codebase"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

Testing is there to answer “Does this code behave correctly?” Verification, on the other hand, answers “Does this code match what we\u2019ve agreed to build?”<\/p>\n\n\n\n

Before AI came into the picture, the human author was also the one carrying the intent, and the two questions collided into one. An AI agent has no intent, however. It just follows the prompt. So if the prompt lacks some details or is a bit unclear, it can end up writing tests that don\u2019t just miss the point, but actually misunderstand something and treat it as correct.<\/p>\n\n\n\n

There\u2019s actually a real-world example of this. In Aviator’s intent-driven experiment<\/a>, a verification agent checked 65 acceptance criteria in 6 minutes and caught 4 failures that weren’t bugs at all. They were intent mismatches, and that’s the class of failure your CI pipeline isn’t built to see.<\/p>\n\n\n\n

TL;DR:<\/strong> Testing code means checking if it runs correctly, and the verification of code confirms that it does what has been agreed upon. They sound the same, and for a very long time, developers and testers actually treated them as the same thing. AI flipped that. Now, when an agent generates 6,000+ lines of code from a single prompt, every test can pass, even though the code is doing the wrong thing. And there\u2019s nothing in your pipeline telling you that.<\/p>\n\n\n\n

Two Similar, yet Different Questions<\/h2>\n\n\n\n

When you run a test suite, you see the following behavior: given X inputs, code produces Y outputs.<\/p>\n\n\n\n

AI code verification asks something different: \u201cDoes this implementation match what was agreed on before writing any code?\u201d Here, it\u2019s not about whether a function returns the right value but whether the function itself is<\/strong> right.<\/p>\n\n\n\n

The reference point here is an external artifact a human has approved (such as a spec with acceptance criteria), not the code\u2019s own logic.<\/p>\n\n\n\n

\"Testing<\/figure>\n\n\n\n

Testing vs. verification<\/em><\/p>\n\n\n\n

The two don\u2019t always line up. An agent misreading a prompt will write the wrong code, as well as the tests that confirm this wrong code works. Green checkmarks all the way!<\/p>\n\n\n\n

As Ankit Jain put<\/a> it<\/a>, tests tell you whether the code does what the author has intended, but they can\u2019t tell you whether that intent was the right thing to build.<\/p>\n\n\n\n

The Author Carried the Intent<\/h2>\n\n\n\n

But how did the industry survive while treating these as one question? The answer is simple: because the intent lived in a human brain.<\/p>\n\n\n\n

When an engineer wrote a feature, they knew what they meant. The ticket was vague, but the gap got filled by the person typing. Code review worked as informal verification because a reviewer could simply ask why something was done in a specific way and get the answer backed by real context.<\/p>\n\n\n\n

An Agent Has No Intent, Only a Prompt<\/h2>\n\n\n\n

An LLM-infused agent doesn\u2019t know what you meant (no mind-reading Neuralink features yet). But, it knows what you wrote, and it sticks to that.<\/p>\n\n\n\n

If the prompt captures 90% of the intent, the agent builds 90% of the feature and fills the rest with probabilistic, yet plausible guesses. Then the agent writes tests for what it has built, including these probabilistic, yet plausible guesses. There is no person to rationalize. You can ask the agent, obviously, but you will just get a confident explanation for whatever it ended up producing.<\/p>\n\n\n\n

It\u2019s not so much that agents write buggy code. Most of what they produce runs and passes tests. The real issue is that \u201ccorrect\u201d and \u201cwhat was agreed upon\u201d split apart as soon as intent gets turned into a prompt. And since generation now outpaces review by an order of magnitude<\/a>, no human has the time to read everything closely enough and actually notice these issues.<\/p>\n\n\n\n

You need a mechanism that compares the implementation against the agreement itself. But this only works if the agreement exists as an explicit artifact, and that\u2019s the core idea behind spec-driven development<\/a>.<\/p>\n\n\n\n

What 65 Criteria in 6 Minutes Looks Like<\/h2>\n\n\n\n

The engineering team at Aviator ran this test end-to-end<\/a>. Engineers wrote and reviewed a detailed spec before implementation, and a human approved it. An agent then built the feature, roughly 6,000 lines, none written by hand. A second agent verified the output against the spec’s 65 acceptance criteria. The verification took 6 minutes: 60 criteria passed, 4 failed, 1 partial.<\/p>\n\n\n\n

The failures were the places where the implementation did something reasonable that was not what the team had actually agreed on. Without that spec as a reference, those criteria would have shipped wrong, only to be discovered later by an unlucky user.<\/p>\n\n\n\n

A human doing the same check would have taken hours. They would probably have done it only once, and their attention would start to drift by criterion 40. The agent, on the other hand, can redo it on every revision.<\/p>\n\n\n\n

Verification has just gone from something that\u2019s almost unaffordable to something you can run on every push.<\/p>\n\n\n\n

Where Testing Fits<\/h2>\n\n\n\n

None of this makes testing optional. It still catches what verification never will: regressions, broken edge cases, performance cliffs. The point is that testing answers a narrower question, and verification is right above it.<\/p>\n\n\n\n

<\/th>Testing<\/th>Verification<\/th><\/tr><\/thead>
Question<\/strong><\/td>Does the code behave correctly?<\/td>Does the code match what was agreed upon?<\/td><\/tr>
Reference point<\/strong><\/td>The code’s own expected behavior<\/td>A human-approved spec<\/td><\/tr>
What it catches<\/strong><\/td>Bugs and regressions<\/td>Intent mismatches and scope drift<\/td><\/tr>
Blind spot<\/strong><\/td>Working code that builds the wrong thing<\/td>Right thing that\u2019s been built poorly<\/td><\/tr>
Written by<\/strong><\/td>Often the same agent that wrote the code<\/td>Derived from the approved spec<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Each layer covers what the other can’t. That’s the Swiss cheese model from Ankit’s Latent Space<\/a> piece: trust comes from stacking layers with different gaps so that holes don\u2019t overlap.<\/p>\n\n\n\n

But in practice, most teams are shipping with one slice while treating it like two.<\/p>\n\n\n\n

What Changes in the Pipeline<\/h2>\n\n\n\n

Most CI\/CD pipelines encode a simple assumption: tests pass, code is ready. Every gate checks behavior, but nothing checks agreement, because this used to be verified by humans. The problem is that the loop now has fewer humans and far more code. (AI-generated code, to be precise).<\/p>\n\n\n\n

Verification becomes its own gate, with the spec as the reference:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

From spec approval to agent implementation<\/em><\/p>\n\n\n\n

Two things change:<\/p>\n\n\n\n

1. A spec exists before implementation.<\/strong><\/p>\n\n\n\n