This one came out of our own operation. We run a market intelligence pipeline that pulls public property sales records out of county sources and turns them into numbers we actually make decisions on. Last week we pointed AI agents at a big chunk of it: a query layer that answers questions like “what did the median sale price do this year” from individual recorded transactions.
It got built fast. It also came back with a green test suite and several confidently wrong answers. The gap between those two things is the whole lesson.
What we were building
Two things were already true. We had one layer that answers from published market time series, the kind of aggregate data a listing site or a federal index puts out. And we had a second body of data that is completely different in character: individual recorded sales, one row per deed, straight from the county.
Those are not the same thing and they cannot be treated the same way. An asking price is not a sale price. A monthly index is not a count of transactions. So we built the second one as its own lane rather than bolting it onto the first, with the rule that neither lane gets to quietly answer for the other.
The first surprise: green does not mean done
The suite came back passing. Seventy tests, all green.
Two of the six metrics were literally this:
return {} as any;Empty stubs. The tests covering them existed and were failing, but nobody had run them, because the test runner’s configuration never reached that folder. Twenty-plus test files across the whole project had never executed a single time. They were sitting in the repository looking like coverage.
That is the first thing worth internalizing if you are having AI write code for you. A test file is not a test. A test that never runs is a comment with extra steps. Before you trust a suite, make the runner tell you which files it actually executed, and go looking for the ones it skipped.
The second surprise: the fixture picked the one shape that could not fail
Once the tests ran, we had an adversarial review pass over the code. It found something we would have shipped.
A deed that conveys several parcels repeats the full sale price on every parcel’s row. Sum those rows naively and you invent money. We knew that, and the code handled it. What the code did not handle was the opposite case: one sale recorded under several different instrument numbers.
In our data, four transactions are like that. Two of them are a single thirty-five thousand dollar sale carrying three separate instruments. The code counted each instrument as its own deed and stamped the full price on each one, then reported that it had caught zero double counting. On this year alone that overstated sales volume by $783,500.
The test fixture had an ambiguous transaction in it. It just happened to give that transaction the same instrument number on both rows, which is the one arrangement of that scenario that cannot expose the bug. The fixture was not wrong exactly. It was agreeable.
This is the failure mode that matters most with generated code. The model writes the implementation and the fixture from the same mental picture, so the fixture confirms the picture instead of testing it. When the same author supplies both the answer and the exam, passing tells you they were consistent, not that they were right.
The third surprise: it invented a plausible fact
Later in the same build, an agent was asked to register geography support only where a data source genuinely publishes postal codes. It registered a real, plausible, correctly formatted ZIP code for the right county.
That ZIP appears zero times in our data. The two that actually appear are different ones. Its own tests passed, because it wrote the fixture using the ZIP it had guessed.
Nothing about that output looks wrong. It looks like local knowledge. The only thing that caught it was running a count against the real database, which took about fifteen seconds. An agent that cannot reach your production data will substitute plausibility for evidence every time, and plausibility is very convincing when it is also well formatted.
What we changed
We did not slow the agents down. We changed what counts as done.
Anything keyed to real data gets a row count before it lands. Registering a ZIP, a county, a source, a metric: query the actual table first, and leave the verified number in a comment next to it so the next person knows it was checked rather than assumed.
A refusal beats a silent zero. An unregistered geography now returns an explicit “not available, and here is why.” A registered one with no rows behind it would have returned zero, and zero reads like a finding. It reads like “there were no sales.” Being told no is safer than being told nothing in a way that looks like something.
The two lanes are not allowed to agree by accident. Where both could theoretically answer the same question, the system refuses to compare them unless a human has written down that the comparison is valid. Two metrics sharing a name is not evidence they measure the same thing, and there is a regression test that fails the day someone introduces an overlapping metric without deciding, in writing, whether the lanes actually mean the same thing.
Every gate has to be shown failing. Before shipping a check that verifies a data load, we ran it against a deliberately broken load and a deliberately incomplete one to confirm it goes red. A gate nobody has watched fail is decoration.
The part that generalizes
None of this is an argument against having AI write your code. The layer got built in a fraction of the time it would have taken by hand, the review that caught the money bug was also an agent, and the sixty tests that did work were genuinely good tests.
The argument is about where you spend your attention. When code was expensive to produce, most of your risk lived in writing it. Now that it is cheap, most of your risk lives in believing it. The verification step is not overhead on top of the real work anymore. It is the work.
The practical version, if you take one thing: for anything that produces a number you would act on, go get that number a second way. Query the database by hand. Compare it to the thing the system says. We did that at the end of this build, and the live system returned a median of $310,000 across 871 qualifying transactions, matching the SQL we wrote by hand, down to the excluded rows. That check took two minutes and it is the only reason we would put the number in front of anyone.