Validation debt
Back in 2023, I asked an AI model to make me a Western blot. Then I asked it to make the blot more realistic. Then more realistic again. The results were comical. The first one was a cartoon, and any scientist who has run a blot could see that AI couldn't make a realistic one. It was the early days of AI.
But buried in that early experiment was a question I keep coming back to. What happens when these models can make an image that doesn't look fake?
That question turned out to be too small. The problem isn't one fake image. And it's not a question about scientific integrity and fraud. It's a question that shows how the cost of producing a scientific "result" is falling toward zero, and the cost of checking one isn't falling at all.
What was hard is now routine
I want to be clear about something first, because it's easy to read the rest of this as a complaint. But it isn't.
A lot of science that used to take a team and several years now takes one person and a week. An analysis that used to need a bioinformatician now runs in an afternoon. A protein that needed a structural biology lab can be predicted before lunch [ed. note: predicted, not solved, but still]. Papers keep getting longer because each scientist can do more, and AI agents (software that carries out multi-step work on its own) push that further every month. More people can do more science. Ultimately, I think that's good.
But every one of those results makes a claim. And every claim needs someone to check it.
Borrowing against the bench
Software engineers have a name for this. In 1992, Ward Cunningham called it technical debt: every shortcut you take to ship faster is a loan, and the interest comes due later, usually at a bad time.
Science has its own version. I call it validation debt. Every result we produce faster than we can validate it is a loan against the bench. The paper is out, the claim is in the literature, other people start building on it. Validation (an independent experiment, a replication, a control someone forgot to run) gets deferred.
And deferred validation compounds. A result that nobody checked becomes the foundation for three more.
We've been here before. In 2011, scientists at Bayer reported that when they tried to reproduce published findings on drug targets in house, only a fifth to a quarter of their 67 projects matched the literature. A year later, Glenn Begley and Lee Ellis at Amgen reported that they could confirm just 6 of 53 "landmark" preclinical cancer studies.
Those weren't fraud cases. They were ordinary results, produced faster than anyone checked them, built upon by everyone downstream. That was the bill for a debt the field didn't know it was carrying.
That reckoning happened with human-speed science. Now imagine the same at the speed of AI-enabled science.
Where the debt shows up
The most common form of validation debt isn't dramatic. It's small, like a missing control or a missing assumption.
Last week the newsletter covered a preprint showing that simple models match large single-cell foundation models on the tasks those models were built for. A foundation model is a big AI model pretrained broadly and then reused for many jobs. The simple baseline is exactly the control a reviewer asks for: does the fancy method beat the obvious one? For a while, the answer was assumed. Last week someone checked.
That's what paying down validation debt looks like. Not a scandal. A validation control.
In 2022, I wrote about Bill Kaelin's plea to "publish houses of brick, not mansions of straw": fewer claims, more proof. Agents make bricks cheap. They don't make inspecting them any cheaper. Further, agents make mansions of straw easier to build than ever.
So what do we do?
There are two bad answers.
The first is to slow down: treat every AI-assisted result as suspect and gate it behind more review. I wrote about the absurdity of that system in A publishing problem. Adding more gatekeepers to a broken gate doesn't fix it.
The second is to ignore the debt and keep producing. That works until the interest comes due.
The better answer is to treat validation as part of the work, not a step after it. A few things follow from that:
- Ship the control with the claim. If you're going to report that a method works, report the simplest thing it beat. If it didn't beat anything simple, that's the finding.
- Make validation cheaper, too. Some of the same tools that produce results can check them. The newsletter recently covered an agent built to audit statistical assumptions before it runs an RNA-seq analysis. That's an agent paying down debt instead of adding to it.
- Keep a record someone else can follow. A result you can't reconstruct can't be validated. We need to reimagine what a lab notebook looks like that makes it easy to follow and validate the claim. [More on that in a future essay].
None of this is new. It's what good scientists have always done. What's new is the ratio. When producing a result was expensive, the cost of producing it kept the volume down to something the field could check. That brake is gone.
The question I keep asking
Which brings me back to the Western blot.
The worry in 2023 was fakes. The real question is bigger: what happens to science when anyone can publish a discovery? I think it's mostly empowering, wonderful, amazing. More people asking more questions with better tools is how science is supposed to work.
But a discovery isn't a discovery until someone validates it. The field's capacity to do that is the scarce thing now, and we're borrowing against it every week.