A practical comparison for leaders evaluating AI-driven document automation
Meta description: Two AI document-processing architectures, tested head-to-head on a real 69-page commercial contract. Here’s what each one is actually good for, where each one fails, and what it costs.
Every enterprise sitting on a pile of contracts, leases, or compliance documents eventually asks the same question: can AI just read these for us? The honest answer is yes — but how you have it read them changes the outcome, the risk profile, and the bill by an order of magnitude. We recently ran two fundamentally different AI document-processing architectures side by side against the same real-world document — a dense, 69-page commercial lease combining legal prose, financial terms, and engineering drawings — to see how they actually compare once you move past the pitch deck.
This isn’t a story about which AI model is “smarter.” Both approaches used capable, current AI models. It’s a story about architecture — what each pipeline is built to look at, what it ignores by design, and why that distinction matters more to your bottom line than any benchmark score.
Two Philosophies for Reading a Document
Approach A: Visual Extraction. This pipeline scans a document specifically for figures, diagrams, and tables, clips those regions out as images, and asks an AI model to describe what’s in each one. Think of it as building a searchable gallery of the document’s visual content — engineering drawings, site maps, equipment layouts. It never reads the surrounding paragraphs; that’s outside its job description.
Approach B: Full-Document Structured Extraction. This pipeline reads the entire document, classifies each section into a logical category (e.g., “lease agreement,” “site application,” “legal description”), and extracts key facts into a structured record — rent, term length, dates, parties, coordinates — with a confidence score and a flag on anything incomplete or ambiguous.
Put simply: Approach A builds an image library. Approach B builds a database. Neither is “better” in the abstract — they answer different questions.
What We Learned Testing Both
The results were clarifying, not close.
Coverage. Approach A processed 7 pages out of 69 — the ones containing diagrams. It has zero visibility into the other 62 pages, including every commercial term in the document. Approach B processed all 69 pages and correctly extracted the core deal terms — monthly rent, escalation rate, lease duration, and site coordinates — verified accurate against the source text.
Where each one struggles. Approach B occasionally lumped unrelated trailing content (in this case, corporate filings and a delivery receipt) into one oversized, mislabeled record — a taxonomy gap, not a comprehension failure; the underlying content was still read correctly. Approach A occasionally labeled visually similar pages inconsistently, and a deduplication step designed to avoid re-processing identical images turned out to sometimes drop distinct low-quality images by mistake — a bug worth fixing, but a narrow one.
The quality of what each produces. Approach A’s image descriptions were excellent — genuinely rich technical detail (equipment specifications, mounting hardware, physical layout) that the full-document pipeline didn’t capture at the same depth. Approach B’s structured extractions were the only output that gave us auditable, reviewable business facts, complete with a signal for which records needed a human to double-check them.
Neither pipeline is “wrong” for existing. They’re built for different jobs, and the test results back that up cleanly: if the deliverable is a searchable index of engineering visuals, Approach A is well-suited to it. If the deliverable is structured, reviewable commercial data, only Approach B can produce it at all.
The Cost Question C-Suite Actually Cares About
This is where it gets interesting, because the cost gap isn’t subtle — and it isn’t where most people would guess.
| Approach A: Visual Extraction | Approach B: Full-Document Extraction | |
|---|---|---|
| Pages actually processed for AI description | 7 of 69 | 69 of 69 |
| Cost driver | Flat per-page scanning fee (charged on every page, whether or not it contains a usable image) | Per-token AI processing (scales with actual content read) |
| Approximate total cost, this document | ~$1.05 | ~$0.02 |
| Cost per page of useful output | High | Negligible |
The roughly 60x cost gap is almost entirely explained by one structural fact: the detection technology Approach A depends on bills by the page scanned, not by the page that yields something useful. It has to look at all 69 pages to find the 7 with diagrams, and it pays the same fee whether a page turns out to be relevant or blank. The AI description step itself — the part everyone assumes is expensive — was actually the cheaper half of that pipeline. Approach B, by contrast, only spends processing budget on content it actually reads and extracts, at a small fraction of a cent per page.
The business implication: Approach A’s cost scales with document length, not with how much of the document is actually useful to you. A 300-page document with the same handful of diagrams costs several times more to run through Approach A, for identical output. That’s a very different cost curve than Approach B’s, and it matters enormously once you’re processing hundreds or thousands of documents a month rather than one.
The Real Lesson: Match Architecture to the Job, Then Price It Accordingly
The instinct to pick “the best AI pipeline” is understandable but slightly the wrong question. The right question is: what decision is this output going to support, and what does it cost to support it at your actual volume?
For most contract- and lease-heavy businesses, the practical answer is a hybrid: use full-document extraction as the backbone for coverage, cost-efficiency, and auditable business data, and reserve visual-extraction technology narrowly — only for the specific pages that actually contain diagrams or drawings, rather than running it as a first pass across an entire document. That single scoping decision alone would have cut the visual pipeline’s cost on this test document by roughly 90%, without losing any of the detail it’s genuinely good at capturing.
Three takeaways worth carrying into any AI document-automation vendor conversation:
- Ask what the pipeline is blind to, not just what it can do. A pipeline that never reads plain text isn’t a weaker version of a full-document tool — it’s a different tool solving a narrower problem. Know which one you’re buying.
- Demand visible confidence and review flags, not just clean-looking output. The pipeline that tells you where it’s unsure is more trustworthy than the one that returns everything with equal, unearned confidence.
- Model your costs against your actual volume and document mix, not a demo. Per-page scanning fees and per-token AI costs scale completely differently. At pilot scale, the difference is a few dollars. At production scale, it’s the difference between a line item and a budget conversation.
The technology to do this well already exists. The competitive edge isn’t picking the flashier pipeline — it’s understanding what each one is actually built to see, and architecting around the gaps.