Polish Without Priors

Think hard, I told the LLM, and asked it to find logic errors in an architectural flow diagram. The response came back in the format of smooth competence: a heading called "Critical Logic Errors," three numbered items, and an explanation under each. Hard to skim past without nodding.

The first item was dead wrong. It described a duplicate POST request (endpoint, parameters, and sequence), none of which was in the diagram. I pushed back: Where specifically do you see that POST? Immediate response: You're right to push back. I misread the flow.

There was a real duplicate-request issue in the flow. The model's intuition was directionally correct. But every specific that mattered was fabricated, and asserted in exactly the same authoritative tone it used after the pushback for the real issue. If I hadn't asked where specifically, I'd either have dismissed the warning (because the endpoint didn't match) or believed the wrong version (because the explanation was structured to be believed). The form of confidence was identical in both cases.

Pure hallucination is the easy case. It's nonsense, and you catch it fast. What bit me here was worse: a claim that's directionally right and specifically fabricated, where the part that's correct lends the invented specifics a credibility they didn't earn.

The bottleneck in software development used to be writing code. We removed it. But that friction was doing work, and what replaced it is a product, built to keep you using it.

People say AI coding tools make you lazy. I don't think laziness is what's going on. The product you're coding with is tuned for a specific outcome, and that outcome is not "you understand this code." It's "you are still here in fifteen minutes." Sometimes those two outcomes line up fine. But they pull apart often enough that the new bottleneck, review, has new pitfalls.

I wrote about AI-assisted coding ten months ago¹ and landed somewhere optimistic. My take then was that the craft was mutating rather than dying, that the hierarchy of skills was inverting toward system design, and that the developers who'd thrive would be the ones who could manage AI agents the way a conductor manages an orchestra. I still think parts of that hold up. But watching it play out has convinced me the "conductor" framing does more to make engineers feel crisply efficient than to keep them honest.

Writing was the review

When you write code yourself, you're doing two things at once. Producing the artifact and reviewing it as you produce it. The act of writing forces you to hold the problem in your head, decompose it, confront the parts you don't understand, and choose between alternatives. By the time the function compiles, you've already reviewed it once.

Generation was never really the bottleneck. Understanding the code was, and still is.

When you accept generated code, the artifact arrives without the cognitive work that would have produced your understanding of it. Now you're reviewing unfamiliar code by an unfamiliar author, except the codebase says it's yours and the PR carries your name. The review pass that used to be free isn't free anymore. It has to be deliberately scheduled and it's now the most expensive part of the process.

AI as a productivity multiplier counts generation speed and ignores review debt. Whether the trade is a net win depends on whether you actually do the second pass. In practice the debt often doesn't get paid by the author at all. It gets transferred to whoever opens the PR next.

Tuned for confidence

That review burden is by design. The product is tuned to produce confidence rather than correctness. Calibrated uncertainty reads as weakness. Confidence reads as competence, and human raters reward it, benchmark scores reward it, and so do the companies whose commercial success depends on you continuing to use the product.

I made this argument once already in a different domain²: you can't reliably detect deceptive alignment, because the system is structurally optimized to pass detection. Same here. You can't reliably get calibrated uncertainty from an engagement-optimized model, because calibrated uncertainty is exactly what engagement optimization selects against.

The people building these tools aren't malicious. But stop treating the thing as a neutral instrument. It's a tuned commercial system whose interests overlap with yours sometimes and run against them continuously.

Processed food got tuned to the bliss point, the flavor mix that overrides the brain's "I'm full" signal. Slot machines got tuned to variable-ratio reinforcement, which is the most habit-forming reward schedule we know of. And cigarettes got tuned year by year for nicotine delivery efficiency. None of those were built by villains either.

Reviewing without priors

Reviewing code you wrote yourself is cheap because you have priors. You know what you were trying to do. You know which choices were intentional and which were hurried. You can sample most of it and audit only the parts that feel uncertain.

Reviewing a colleague's code costs more but stays manageable, because you've built priors on that colleague: how they think, where they cut corners, and what their mistakes usually look like. A questionable pattern from a trusted author reads as intentional and prompts a question, not a correction. The review economy of trust compresses the work.

Reviewing an open-source dependency is different again. There's no individual to trust, but there's a process: versioning, changelogs, issue trackers, semver discipline, and public scrutiny. The trust is institutional rather than personal.

AI-generated code has neither. There's no individual history behind it and no institutional process around it. And yet it shows up wearing the surface signals of both, with consistent style, idiomatic phrasing, plausible structure, and the polish of code that already went through review before it got to you. So it deserves the scrutiny you'd give a stranger's first PR, and instead it gets what you'd give a trusted colleague on their tenth.

The author is supposed to be the first reviewer. That only works if the author treats the code as unfamiliar, which is hard, because sitting there while it generates feels like writing it. When the author skips that, the next person to open the PR inherits the cost.

On capability

The reasonable counterargument is that all of this is transitional. Models improve. Verification tooling matures. Agentic loops will eventually self-review with more discipline than tired humans bring.

Some of that is likely right. Calibration is itself a research target now. Anthropic and others publish on it, and the next generation of models may genuinely know what they don't know in a way the current ones don't. Agentic harnesses are getting structured review baked in: second-model audits, formal checks gating merges, evals that score honesty as well as capability. The review tenets that read as paranoid today will look like ordinary engineering practice in three years.

What I don't expect to change is the incentive structure underneath. As long as the dominant business model is tokens-sold-per-session, the optimization pressure on these models keeps pointing at fluency, engagement, and continued use. A more accurate model can still be tuned to keep you in the chat. We've watched this happen before, with search, with social media, with the recommendation feeds that decide what you see next. The B2B framing maybe pushes the other way (enterprise seats sold to procurement teams that care about throughput, not engagement), but individual engineers pick their own tools, and the consumer-grade incentives leak in through that door.

You can think capability will keep climbing and still worry about where the incentives point. Those are separate claims. Capability is a moving target. But the argument in this post doesn't expire when GPT-7 ships, because GPT-7 will still be sold by the token.

The discipline

The hard part, if you accept all this, is that there isn't a clean fix.

Push back with specificity. Where exactly do you see this? Most of the time the answer is: nowhere. The model couldn't sustain its own claim under direct interrogation, and you've just verified that the confidence was form, not substance.

Generate less at a time. The unit of trust is whatever you can hold in your head. If a chunk is bigger than that, it's too big to merge.

Treat your own AI-generated code as unfamiliar code. Sitting there while it generates isn't authorship. Review it like a stranger wrote it, because in any meaningful sense, one did.

Trust the felt sense of this is off. That feeling isn't imposter syndrome. The model's confidence will try to override it, and the override is the trick. When something doesn't click, slow down. Ask the model to walk through the line that bothered you. Most of the time, the gap surfaces within two or three exchanges.

None of this fixes the underlying dynamic. It's a discipline, in the older sense of the word: a thing you have to keep doing because the system you're working with won't do it for you. The model isn't broken when it gets things wrong. It's doing exactly what it was built to do, which is sound convincing, and sounding convincing is a different job from being correct.