A few weeks ago I reviewed a data archiving strategy for a client project — the kind of document that quietly decides how a company will delete years of production data from its primary databases. It was well-organized, technically fluent, and confidently wrong in ways that only surfaced once someone actually tried to break it.
It hadn’t been written by someone careless. It had been compiled by an engineer with limited experience, working with AI to move fast — and the result was a document that sounded like it was written by someone who understood the system deeply, when what it actually contained was a happy path with the rough edges smoothed over.
The more I looked at it, the more I recognized the pattern from somewhere else entirely: code review. AI-generated code fails in almost exactly the same shape as AI-generated architecture docs — it’s fluent, it runs, and it quietly assumes nothing ever goes wrong. This isn’t really a documents problem or a code problem. It’s what happens any time a solution is purely AI-generated and nobody with real domain judgment pressure-tests it before it ships.
The document read well. That was the problem.
Good technical writing and correct technical writing are not the same thing, and AI is extremely good at the first one. Sentences like “dropping a partition is a fast metadata operation” or “the two partitioning schemes are compatible” read with total confidence. They’re also the two claims in the document that turned out to be wrong or unverified once someone actually pressure-tested them.
That’s the pattern to watch for: AI-generated technical prose hedges where the topic is naturally uncertain, and states flatly where a domain expert would actually be most cautious. It doesn’t know which claims are load-bearing, so it doesn’t flag them any differently than routine ones. A human author who deeply understood the system would have written “we need to validate this” next to the two sentences that actually mattered. This document didn’t, because nothing in the writing process forced that judgment to happen.
Four gaps that show exactly where “happy path” thinking crept in
1. The safety mechanism didn’t survive translation from prose to pseudocode. The document’s narrative description of the process was correct: every partition gets copied, then validated, then — only after validation passes — becomes eligible for deletion. The pseudocode included later in the same document quietly broke that guarantee: partitions that arrived through a separate, pre-existing pipeline skipped the validation step entirely, because the loop that ran the checks only iterated over freshly-copied data. Nobody caught it, because nobody traced the logic end to end — each section was internally plausible, and the inconsistency only existed between sections.
This is a distinctly AI-shaped bug. A generated document assembles each section to be locally coherent; it doesn’t naturally cross-check that the pseudocode in section five still enforces the principle stated in section one.
2. A widely-repeated technical claim went unverified. The plan relied on database partition drops being a low-impact, near-instant operation. That’s true in some configurations and false in others — under real production load with a large buffer pool, it can stall the database entirely. The claim wasn’t fabricated; it’s a genuinely common simplification. But it was written into a production runbook as settled fact rather than something to load-test first, because there was no step in the process that asked “have we actually confirmed this against our own environment, or are we repeating something that’s usually true?”
3. Compliance and regulatory context was invisible to the document, because it wasn’t in the prompt. The system in question handled financial transaction data in a regulated industry, subject to retention obligations that have nothing to do with whether a BI dashboard queries the table. The document scoped what to keep purely by reporting usage. That’s not a technical failure — it’s a domain failure. AI tooling will faithfully solve the problem it’s given; it has no way of knowing a regulator cares about data nobody’s dashboarding needs, unless a person with that context puts it in scope.
4. The safety net had no safety net of its own. The entire design rested on an archived copy being trustworthy enough to justify deleting the original. Nothing in the document protected that archive from being overwritten or deleted itself — no versioning, no immutability, no backup story. It’s an easy thing to miss when you’re deep in designing the forward path (copy, validate, delete) and never stop to ask what protects the thing everyone’s now trusting.
The same pattern, written in code
Swap “document” for “pull request” and almost every one of those four gaps has a direct equivalent in AI-generated code that hasn’t been pressure-tested by someone with domain judgment:
The tests confirm the code does what it does, not that it does what it should. AI-generated test suites are very good at asserting that a function returns the expected value for the input it was designed around, and much less reliable at generating the failure case nobody thought to ask for — the malformed payload, the network timeout mid-write, the second request that arrives before the first one finishes. A green test suite feels like proof of correctness. It’s often just proof that the happy path works, written by the same process that wrote the happy path.
Error handling that looks thorough often isn’t. A try/except around a risky operation reads as defensive code. Whether it actually is defensive depends on what happens inside the except block — and AI-generated code has a strong tendency to log the error and continue, or return a default value, rather than propagate a failure that genuinely needs to stop the process. It’s the code equivalent of the document that mentions a risk in its “risks” section and then never actually gates anything on it.
Local coherence, global inconsistency. The exact bug that broke the archiving document’s safety guarantee — a validation step that one part of the system assumes happens and another part quietly skips — is one of the most common shapes of AI-generated code defects. Function A calls Function B assuming it enforces a constraint; Function B was generated later, in a different context, and doesn’t actually enforce it. Each function is individually reasonable. The system isn’t.
Performance and safety claims baked in without being verified. “This operation is idempotent,” “this query is indexed,” “this is safe to retry” — these show up as comments or naming conventions in AI-generated code with the same unearned confidence as “dropping a partition is fast metadata operation” showed up in the document. They’re often true. The problem is never checking which ones are false for your actual data volume, your actual concurrency, your actual failure modes.
Nobody encoded the constraint nobody mentioned. Just as the archiving document never considered regulatory retention because nobody told it to, AI-generated code faithfully implements exactly the requirements it’s given — and stays silent about the security check, the rate limit, the authorization boundary that a domain expert would have flagged automatically because they’ve been burned by its absence before.
The same pattern, in a vendor’s cloud migration proposal
A separate example, unrelated to the archiving project: an AI-assisted technical and cost proposal a vendor sent to a cloud partner ahead of a migration. It described several AI product use cases, an architecture, and a detailed monthly cost model — the kind of document that gets quoted back as a budget once someone signs off on it. Same pattern, three more instances of it.
A design principle that didn’t survive to the next section. One section stated a firm rule, twice: a smaller, cheaper model was explicitly excluded from any user-facing interaction, because the product’s whole credibility depended on higher-quality reasoning at that point. Two sections later, in the architecture for a different but still real-time, user-facing feature, that same excluded model reappeared running half the workflow. Each section was internally consistent on its own. Nobody checked whether the principle stated in section one still held once section three’s architecture was written — the identical seam-blindness that let the archiving document’s pseudocode quietly drop its validation guarantee.
A safety net for the thing the whole product’s credibility rests on — that didn’t exist. The product’s core value proposition rested on the AI citing its sources precisely enough that a professional could verify the answer. But nothing in the design verified that what the model claimed about a source actually matched the source itself. The only quality control mentioned anywhere was manual spot-checks, with automated verification deferred to “when it matures.” Nothing flagged this as a risk — it was simply absent, the same way the archiving document never considered regulatory retention because nobody had put it in scope.
Confident, specific-sounding numbers that hadn’t been checked against the rest of the same document. The cost model contained per-service estimates precise to the low hundreds of dollars — a token-usage figure, a managed search-service figure, a document-extraction figure — each stated as a number to plan a budget around. Rebuilding the math from the product’s own stated technical requirements (context window size, document sizes, which processing tier a stated feature actually needed) put several line items off by anywhere from 2x to 20x. Each number was individually plausible. Nobody had cross-checked it against the same document’s own product description two pages earlier.
None of this points to a careless author. It’s the predictable output of a process that writes each section to be locally complete, with no mechanism for checking whether section one’s stated principle, section three’s architecture, and section seven’s cost table still agree with each other.
What actually changes: the review, not the tooling
None of this is an argument against using AI to write documents or code — it’s genuinely good at both, and refusing to use it doesn’t make the underlying risk go away. What changes is what a review has to look for, whether the artifact is a Word document, a pull request, or a client-facing proposal:
- Trace claims and assumptions across the whole artifact, not section by section or function by function. Contradictions hide exactly at the seams between parts that were each individually generated to sound complete.
- Flag every sentence or line that states a performance, safety, or correctness property as fact, and ask: has this actually been verified here, or is it a plausible generalization the model has seen a thousand times?
- Ask who’s missing from the room. Compliance, security, legal, and operational context don’t show up in AI output unless someone with that context supplied it — the absence is silent, not flagged.
- Find the thing the whole design implicitly trusts — the archive, the retry, the assumption that a check upstream already happened — and check who’s actually protecting it. Every system has one, and it’s the thing most likely to be assumed safe by default.
- Read the failure paths before the success paths. In both a document and a codebase, the happy path is where the least judgment was required to write it, and the least value is added by reviewing it again.
Treat AI-generated technical output — prose or code — the way you’d treat a first draft from a smart, capable person who’s brand new to your business and has never seen an incident caused by the thing they just wrote with total confidence. Fluent, useful, structurally sound — and in need of someone who’s actually been burned before to go looking for exactly where the happy path was quietly assumed instead of proven.