The demo always uses the same document: a native PDF, generated by another system, cleanly laid out, in one language, tables squared off, logo in the top left. The tool reads it in two seconds and returns every field. Applause. And it is true, there is no trick: any decent product on the market handles that document today, and handles it cheaply. The problem is that document is not yours. Or more precisely: it is 85% of yours, and the 85% was never the project.
The thesis in one line: in document extraction, the budget and the calendar are not set by the easy part. They are set by the 15% of odd documents no demo ever shows — the crooked scan, the table split across two pages, the handwritten delivery note with a stamp over it, the supplier who changed their template in July without telling anyone. And the number they sell it with, average accuracy, is precisely the metric designed to keep that tail out of sight.
AI document data extraction errors are not spread evenly
When a vendor tells you "97% accuracy", what you are hearing is an average. And an average, by definition, spreads: it suggests three documents in every hundred have some minor problem, scattered at random, like losing three socks a year. That is not how it works. Extraction errors do not sprinkle themselves over the pile. They cluster, and they cluster in the same places every time. The same handful of suppliers, the same kind of scan, the same cursed field.
That clustering is what turns a pretty number into a long project. If the 3% failure rate really were spread out, the process absorbs it: somebody fixes three loose things and moves on. If the 3% is always the documents from the same fourteen suppliers, you do not have a 3% error rate — you have fourteen suppliers still being keyed in by hand, with their own parallel flow, their own dedicated person and their own spreadsheet. The savings you budgeted evaporate there, not in the model.
The general logic — why an accuracy percentage is not enough in production and how the threshold is set by the cost of the error — we covered separately. Here we get specific: what exactly those documents are, why they break, and what you need to demand before you sign anything.
The five documents that eat the project
These are not five exotic edge cases. They are the five that show up in any real operation the moment you have spent two weeks looking at the actual inbox instead of the folder of examples somebody curated for the meeting.
| What arrives | Why it breaks | What it actually costs |
|---|---|---|
| The phone photo and the 150 dpi scan | Skewed text, a shadow across half the page, resolution below what recognition needs to tell a 3 from an 8 | The error is invisible: the amount goes in, it is plausible, and nobody catches it until close |
| The table split across two pages | The system reads each page as a unit; the last line of the first and the first of the second get dropped or duplicated | The total reconciles and the line items do not, which is exactly what cost accounting needs |
| The handwriting, the stamp and the pen note | The received quantity corrected by hand on the delivery note is the real number, and it is the only one not printed | The system returns the printed figure with full confidence: silent error, the most expensive kind |
| The supplier who changed their template in July | What worked on a template stops working with nobody touching anything, and the system does not flag it because it still finds fields | Weeks of bad data before anyone spots the pattern; the cost is not reprocessing, it is losing trust |
| The ambiguous document: credit note, copy, proforma | The problem is not reading it, it is deciding what it is. The shape is identical to an invoice and the meaning is the opposite | A credit note posted as an invoice gets paid twice; you find it at reconciliation, months later |
Look at the third column, the one nobody reads during procurement. None of the five fails loudly. A system that falls over and returns an error is cheap: you see it, you send it to the manual queue and you carry on. The five above return data — plausible, correctly formatted, with the usual confidence. That is the failure mode that really sets the project calendar, because you do not fix it with a better model. You fix it with validations, business rules and a person watching in the right place.
Why average accuracy hides exactly the expensive part
The arithmetic of why 95% per-field accuracy can be a 60% valid-document rate we already took apart when comparing classic OCR and multimodal models for reading documents: it is measured per field and it is paid for per document, and the field that fails is never the postcode. What matters here is the consequence, which is a budget question rather than a measurement one: the gap between those two numbers is not statistical noise, it is the tail of hard documents turned into permanent manual work.
And this is not a hunch from a meeting room: it is the bottleneck the industry research has been measuring for years. In The State of ePayables 2025, Ardent Partners puts the average invoice exception rate at 18.4% — nearly one invoice in five falls out of the automated flow and ends up in a person's hands — and states explicitly that exceptions are the single biggest reason the rest of the department's metrics do not improve. In the same study, top-quartile organisations are not processing different invoices: their exception rate is 47% lower and they process 1.8 times more invoices straight through than everyone else. The gap between a good AP department and a mediocre one is not in the easy document. It is entirely in the tail. (Scope: a predominantly North American sample of accounts payable professionals; the figure sizes the problem, it is not a result of ours.)
Translated into budget: if you size the project on the 85% that goes through on its own, you are budgeting the part that was already cheap. The real bill comes from the other tail, and it arrives as weeks of calendar, not as licence fees.
What to ask the vendor before you sign
Five questions. None of them technical, all of them uncomfortable — which is exactly the sign that they work. A vendor who answers these well is worth more than one with a flawless demo.
- "Give us your rate on our two hundred worst documents, not on your benchmark." It is the only question that matters. You pick the sample: the bad scans, the odd suppliers, the handwriting. If the answer is that they need to train on them first, that is a fair and honest answer — but now you know there is a project phase that was not in the quote.
- "Give me accuracy broken down by field, not the global number." Global is marketing. Per field you see immediately whether the amount and the tax ID hold up, or whether the average is being propped up by the date and the postcode.
- "What does the system do when it is not sure?" The right answer is that it flags it and sends it to review. If the system always returns a value, it has no notion of uncertainty: it is equally confident about what it knows and what it invents, and that is the silent error from the table above.
- "How do I find out a supplier has changed their template?" Look for an answer with a concrete mechanism — an alert on dropping confidence or on drift in the distribution of values. If the answer is "you would see it in the data", the translation is that you will see it late.
- "Who pays for reprocessing, and on what deadline." Not to punish anyone: to find out whether the cost of the 15% stays on your side of the table. It almost always does. Better to know that before signing than at the second month-end close.
One warning about the first question: build the sample before the meeting, not after. Two hundred documents picked by you, with the ground truth written down by hand. It is a boring day of work and it is the only thing that turns a purchase decision into a measurement.
How to budget this so it actually ships
The approach that works is counterintuitive: you budget the hard tail first and barely budget the easy part at all. In practice that means three decisions taken before a line of code gets written. One, which fields are critical and which are comfortable — the amount gets validated against the purchase order, the postcode gets validated against nothing. Two, where the person sits: not reviewing everything, which kills the savings, but precisely on the documents the system flags as doubtful and on the fields that cost money. That is designing the human in the loop of an automation applied to the most literal case there is. And three, what happens downstream to a document that could not be read, because an extraction project that leaves a hole in the next step has automated nothing: it has moved the bottleneck, which is the classic trap in automating a process end to end with AI.
With those three settled, the technology choice becomes almost secondary — and when it does come up, the comparison between classic OCR and multimodal models for reading documents is covered separately. We build exactly this in the two cases that repeat most: extracting the data from invoices and classifying incoming documents so each one lands where it belongs. In both, the serious work is not reading the pretty PDF. It is deciding what gets validated, who looks at the doubtful ones and what happens when the ugly document arrives.
The conclusion, unvarnished
The question almost everybody buys on — "what accuracy does it have?" — is the wrong question, and not because the number is false, but because it describes the part of the work that was already solved. The useful question is what your system does on the day the handwritten delivery note, the split table and the big supplier's new template all arrive. If the answer is "it flags it, sets it aside and tells someone", you have a project. If the answer is a percentage, you have a demo.
Do not buy average accuracy. Buy behaviour in front of the ugly document.