OCR or LLM for document data extraction: which one you need (and why the hybrid wins)

The same email lands almost every week: «we get thousands of documents a month and we're keying them in by hand — should we use OCR or AI?». It sounds like a technical question and it's really a procurement one: somebody has two quotes on the desk, one cheap and predictable, one flexible and expensive, and wants to be told which. The honest answer is a little uncomfortable: the technology doesn't decide this. The shape of your documents does.

The thesis in one line: this isn't about AI yes or no. It's stable template vs chaotic format, and what decides in the end is cost per document. If your documents always have the same shape, classic OCR is cheaper, deterministic and auditable. If every one of them shows up however it likes, a multimodal LLM reads anything with no template — and bills you on every page. That's why in production neither one wins alone: the hybrid wins.

OCR or LLM to extract data from documents: what each one actually does

Classic OCR turns the image into text, and on top of that a layer of rules decides which text is which field: the invoice number lives in the box top right, the total is whatever follows the word TOTAL. The intelligence isn't in the reading: it's in the template. You give it a zone or a pattern and, on the same input, it hands you exactly the same thing every time.

A multimodal LLM doesn't read zones: it looks at the whole page —text, tables, stamps, a handwritten note, the crooked photo somebody took with a phone— and returns the fields you asked for because it understands what an invoice, a delivery note or a contract is. There's no template to maintain. There's also no guarantee two runs on the same document come back character for character identical.

The difference that matters isn't which one is smarter. It's what happens the Tuesday a supplier redesigns its invoice without telling anyone.

Classic OCR fails when you change the shape. The LLM fails when the document is ambiguous and it fills the gap with something believable. Two different failure modes, two different safety nets. What neither of them does on its own is tell you it just failed: you build that part or it doesn't exist.

When to pick classic OCR with a template

Pick classic OCR when the document always has the same shape and that shape doesn't depend on anyone outside your walls. That's everything your own system generates, plus a short list of stable issuers: the same delivery note from the same carrier, the same form your customers fill in, the same bank statement month after month.

• High volume, short list of formats. Few issuers, many documents: the template pays for itself every day that passes.
• You need the same result every time. A process that gets audited, or that triggers a payment, can't depend on today coming out like yesterday.
• You have to explain where each field came from. With rules you point at the zone and the pattern; with a model you point at a paragraph of prose, which is not the same thing in front of an auditor.
• Cost per document has to stay flat. If you're multiplying volume next year, a cost that grows per page shows up far sooner than you expect.

The price of this option is paid in maintenance and it never shows up on the software invoice: every new issuer is a new template, and every redesign someone else makes is a broken one. If your issuer list grows on its own, you're not buying a cheap technology — you're creating a permanent job for somebody.

When to pick a multimodal LLM

Pick an LLM when the shape is the problem. Documents from hundreds of different issuers, contracts where the number that matters lives in a clause each firm words its own way, emails with attachments arriving as PDFs, as photos and sometimes forwarded three times. There a template isn't expensive: it's impossible.

• Every document comes from somewhere else. The cost of maintaining a template per issuer beats the cost of reading with a model, and it never ends.
• The field isn't a zone, it's an idea. «Is there a late-delivery penalty?» isn't in a box: you have to understand the text to answer it.
• Volume is low or lumpy. With a handful of documents a month, building and maintaining templates is a project that never pays back.
• You're still exploring. Before industrializing anything, a model tells you in a week which fields are genuinely extractable and which ones fight back. It's the cheapest reconnaissance there is.

The price here is the mirror image: you pay per document, every month, forever, and you also pay for validation. A model that returns an almost-correct IBAN is worse than one that returns nothing, because «almost» sails straight through every check that only looks at whether the field is filled.

Why the hybrid wins in production

Once a system like this has been live a few months, almost all of them land on the same architecture: some because they designed it that way, the rest because they got there the hard way. It isn't a lukewarm draw between two options. It's the only way to have OCR's cost and the model's coverage at the same time.

1. Classify first. Before extracting anything, the system decides what kind of document this is and who sent it. That step is cheap and it conditions everything downstream — it's exactly the job of classifying the documents that come in.
2. The known bulk goes down the cheap path. If the document matches a template you already have, rules extract it. That's most of the volume, and processing two thousand costs roughly what a thousand costs.
3. The odd stuff goes to the model. New issuer, format that doesn't match, bad scan: that's where the LLM earns its keep, because it doesn't need to have seen the document before. You pay the high price on the tail only, not on the whole catalogue.
4. Cross-checks on meaning. The model is also good for reviewing what the rules pulled out: do the line items add up to the total? Is the due date after the issue date? Does this supplier exist in your master data? Those are errors a template never catches, because it read a wrong value perfectly.
5. A confidence rule that decides who looks. Every field comes out with a score; below the threshold, the document goes to a review queue instead of into the ERP. That threshold is a business lever, not a technical detail: raising it costs people-hours, lowering it costs errors that surface late.
6. Everything a person corrects goes back into the system. A correction that repeats on the same issuer is the signal that this issuer has earned its own template. That's how the hybrid gets cheaper on its own: the expensive tail keeps turning into cheap volume.

This is the underlying pattern of any process automated end to end with AI: the happy path down the cheapest route, the exception down the most capable one, and an explicit door to a human. In document extraction it's almost brutally visible, because cost per document puts everything in black and white.

What reading a document actually costs

The budget line is the small part. The real cost of an extraction system spreads across five places and only one of them appears in the vendor's proposal:

• Processing. What it costs to run the document through the engine. Flat with OCR, proportional to pages and length with an LLM.
• Format maintenance. New templates and broken templates. Effectively zero with a model; grows with your issuer list with rules.
• Human review. What you pay for every document somebody has to look at. Usually the biggest line and almost never in the comparison they show you.
• Reprocessing. Documents you have to run again because they came in truncated, got uploaded twice, or the batch died halfway.
• The error that slips through. A misread amount that reaches accounting isn't paid in software dollars: it's paid in a duplicate payment, a refund, or a close that slides. It's the most expensive one and the only one with no line of its own in any budget.

Before you sign anything, put those five lines on the same sheet for both scenarios, with your real volume and your real issuer mix. It's the same exercise as calculating the ROI of an AI automation: if the saving only appears when you ignore human review, the saving isn't there.

How to measure quality: per-field accuracy, not global «accuracy»

This is where half the projects go under. A vendor shows you a global accuracy number and that number means nothing, because it averages fields that aren't worth the same. Misreading a supplier's trading name is an annoyance; misreading the amount is money moved wrong. A single number blends them and calms you down exactly where you should be nervous.

Measurement that actually helps looks like this:

1. One number per field, not one for everything. Accuracy on amount, on date, on tax ID, on document number. Each with its own bar, set by what it costs to get that one wrong.
2. Exact match, not close enough. On an IBAN or a tax ID, «almost» is wrong. Compare character by character against a set of documents labeled by hand by someone on your team, not against the vendor's demo.
3. Separate «I don't know» from «I made it up». An empty field is a review case; a filled and false field is an incident. A system that goes quiet when it's unsure is worth more than one that always answers.
4. Measure by issuer too. The average hides the supplier whose format has been breaking for three weeks with nobody noticing.
5. Re-measure whenever anything changes. New model, new vendor version, template touched: you run the same test set again. Without that you don't know whether you improved or fixed one thing and broke another.

With those numbers in hand the conversation changes register: you stop comparing demos and start comparing results on your documents, which are the only ones you're going to run.

What to do with human review

Human review isn't the failure of automation: it's the piece that makes it shippable. The mistake is how it gets built. Putting a person in front of a hundred percent of the output means you swapped typing for reading, and reading screens full of correct data is something humans are terrible at: twenty minutes in, everything gets approved unlooked-at and the control exists only on the org chart.

What works is the opposite: the system decides who gets shown what. The person only sees what the confidence threshold flags as doubtful, what exceeds a certain amount, and what comes from an issuer never seen before. And it arrives with the field highlighted on the image, not as a blank form to fill in with the PDF open on the side. That's exactly the argument in human in the loop in an AI automation: the value isn't in supervising, it's in deciding what gets supervised.

When this isn't worth automating

There are cases where the right answer is don't, and saying so out loud saves entire projects:

• Low, stable volume. If the work fits in an hour a week, the project —integration, testing, maintenance— costs more than the problem. Don't automate it: write it down and move on.
• Documents that already arrive structured. If the issuer can send you the same data over an API, as XML or as an e-invoice, reading a PDF is self-harm. Ask for the data properly before you build a reader.
• Nobody knows what the correct answer is. If your own team argues about what «the date» of a document is, no technology settles that argument. Decide first, automate second.
• The document triggers something irreversible. Extract to propose, yes. Extract to execute a payment, a termination or a ruling with no review, not yet.
• The documents are bad at source. Cropped photos, unreadable scans, upside-down pages. Fixing capture usually returns more than any model, and costs a lot less.

That filter is the same one you apply when deciding which processes to automate with AI: repetitive, digital input, a defined definition of correct, and someone who owns the outcome. Miss the fourth and you don't have a project — you have an experiment with a budget.

The decision, in three questions

1. How many distinct issuers generate most of my documents? Few and stable, start with templates. Many and shifting, start with a model.
2. What does a misread field nobody catches cost me? If the answer is «a duplicate payment», your confidence threshold runs high and your human review isn't optional.
3. Can I hand-label a hundred real documents this week? If you can't, you can't measure; and without measuring, choosing between OCR and an LLM is expressing a preference, not making a decision.

The question you arrived with —«OCR or LLM?»— is a catalogue's question, not your operation's. Yours is a different one: which share of my documents has a stable shape, which share doesn't, and what does each cost me. Answer that and the design falls out on its own: rules for the known, a model for the odd, a confidence number that decides when a person steps in, and a counter that tells you whether the expensive tail is shrinking. The rest is picking a logo.