Document AI in 2026: multimodal LLMs, OCR, or both?

Hand a modern multimodal model a scanned invoice and ask for the supplier, the total and the line items as JSON, and it will usually get them right. That is a real change from a few years ago, when the same job needed an OCR engine, a layout model, templates per supplier and a lot of regular expressions.

“Usually” is the problem. When a model gets a document wrong, it rarely says so. It returns a well-formed JSON object with a plausible invoice number that is not on the page, or two columns swapped, or twelve of fifteen line items and no sign that three were dropped. In accounts payable, insurance or lending, that error goes straight into a system of record.

This guide covers what makes documents hard, how classic OCR and document-processing services compare with multimodal LLMs as of September 2026, and the hybrid pipeline we build when the extracted data has to be right: schemas, validation, confidence routing, business-rule cross-checks, evaluation and cost.

What makes documents hard

“Document AI” covers very different problems. A clean, digitally generated PDF invoice is close to solved. A faxed, stamped, partly handwritten claim form is not. Before choosing an approach, sort your documents by the traits below, because each one breaks extraction in a different way. Public benchmarks such as OmniDocBench now test parsers across many document sources and layout types, which is useful context, but they cannot tell you how a tool will do on your suppliers' invoices.¹

Three approaches

Classic OCR and intelligent document processing services

The three big clouds each sell a document service that combines OCR, layout analysis and pretrained extractors for common document types. Their strengths are predictable: word-level bounding boxes, per-field confidence scores, stable per-page pricing, and no tendency to invent text that is not there. Their weakness is flexibility. A field the pretrained model does not know about needs a custom model, queries, or code.

• Amazon Textract detects printed and handwritten text, forms, tables, layout, signatures and answers to natural-language Queries. AnalyzeExpense is built for invoices and receipts and normalises field names across documents. Each item comes with a confidence score from 0 to 100 and bounding box coordinates. Handwriting, invoices and receipts, and Queries are English-only.²
• Google Cloud Document AI offers Enterprise Document OCR (200+ languages, including handwriting), Form Parser, Layout Parser, a pretrained Invoice Parser, and Custom Extractor, several versions of which are powered by Gemini models. Generative extraction in Custom Extractor officially supports English only.³
• Azure Document Intelligence, now branded Azure Document Intelligence in Foundry Tools, has Read and Layout models, prebuilt models such as prebuilt-invoice and prebuilt-receipt with Microsoft-maintained field schemas, and custom models.⁴ The Layout model can return Markdown with tables as HTML, which is a convenient input for an LLM, and returns per-word confidence. It processes PDFs of up to 2,000 pages.⁵

Multimodal LLMs reading the page directly

The major model APIs accept PDFs directly and look at each page as an image, usually alongside any text embedded in the file. Claude converts each page to an image and supplies the extracted text with it; text alone typically costs 1,500 to 3,000 tokens per page, plus image tokens. A request can include up to 600 pages, or 100 when the context window is under 1M tokens, and dense PDFs can fill the context before reaching that limit.⁶ Gemini counts each page as 258 tokens, accepts files up to 50 MB or 1,000 pages, and scales large pages down to 3072 by 3072 pixels.⁷ OpenAI's API sends both extracted text and page images to vision-capable models, with a 50 MB limit per request.⁸

The strength is reading by meaning. An LLM handles a new supplier layout, a table without headers, or a field described in plain English (“the date the goods were delivered, not the invoice date”) without any training. The weaknesses are the ones in the introduction: no reliable per-field confidence, occasional invented values, and errors that look exactly like correct answers.

Hybrids

In a hybrid, an OCR or layout service produces text with coordinates and confidence, and an LLM does the interpretation, seeing both the OCR output and the page image. The OCR layer gives you something to check the LLM against. If the model returns an invoice number, you can confirm that string appears in the OCR text and know which page and position it came from.

Our default for documents that feed a financial or regulated system is the hybrid. For high-volume, low-stakes reading, such as routing inbound mail or summarising contracts for search, an LLM on its own is often enough. For a single, stable form type at very high volume, a pretrained or custom IDP model can still be the cheapest and most predictable choice.

The pipeline we build

1. Ingest and check quality. Reject or flag pages below a minimum resolution, password-protected files and empty pages before spending money on them.
2. Classify and split. Separate invoices from credit notes, delivery notes and statements. Most extraction errors in bundles start as classification errors.
3. OCR and layout. Get text, tables, coordinates and confidence from an IDP service.
4. Extract to a schema. An LLM reads the layout output and the page images and fills a strict JSON schema, including an evidence quote and page number for key fields.
5. Validate. Types, formats, arithmetic and completeness, in code.
6. Cross-check. Against the vendor master, open purchase orders, goods receipts and previous invoices.
7. Route. Straight through, or to a human review queue with the reason attached.
8. Post and learn. Write to the ERP as a draft or for approval. Every human correction becomes a labelled example for the evaluation set.

Structured output, then validate anyway

Every major API now supports schema-constrained output. OpenAI Structured Outputs guarantees that responses follow your JSON Schema, and its Python SDK accepts a Pydantic model directly.⁹ Claude's structured outputs do the same, with a parse() helper for Pydantic models.¹⁰

A schema guarantees shape, not truth. Both providers document the cases where you do not even get the shape: a response cut off at the output token limit may be incomplete JSON, and a refusal may not match the schema.⁹¹⁰ Claude's implementation also rejects some JSON Schema features, such as numeric minimums and string lengths, which its SDK moves into descriptions and checks locally.¹⁰ So we keep the business validation in our own Pydantic model and run it regardless of provider:

from datetime import date
from decimal import Decimal
from pydantic import BaseModel, Field, model_validator

class FieldEvidence(BaseModel):
    field: str                 # e.g. "invoice_number"
    page: int
    quote: str                 # exact text copied from the page

class LineItem(BaseModel):
    description: str
    quantity: Decimal
    unit_price: Decimal
    amount: Decimal

class Invoice(BaseModel):
    supplier_name: str
    supplier_tax_id: str | None
    invoice_number: str
    invoice_date: date
    currency: str = Field(description="ISO 4217 code, e.g. EUR")
    po_number: str | None = Field(description="null if no PO number is printed")
    subtotal: Decimal
    tax_total: Decimal
    total: Decimal
    line_items: list[LineItem]
    printed_line_count: int = Field(description="number of line rows printed on the document")
    evidence: list[FieldEvidence]

    @model_validator(mode="after")
    def check_invoice(self):
        problems = []
        cent = Decimal("0.01")
        if len(self.currency) != 3 or not self.currency.isupper():
            problems.append("currency is not an ISO code")
        if len(self.line_items) != self.printed_line_count:
            problems.append("line items missing or extra")      # silent truncation
        for i, li in enumerate(self.line_items):
            if abs(li.quantity * li.unit_price - li.amount) > cent:
                problems.append(f"line {i + 1}: qty x price != amount")  # swapped columns
        if abs(sum(li.amount for li in self.line_items) - self.subtotal) > cent:
            problems.append("lines do not sum to subtotal")
        if abs(self.subtotal + self.tax_total - self.total) > cent:
            problems.append("subtotal + tax != total")
        if problems:
            raise ValueError("; ".join(problems))
        return self

Two more checks run after parsing. First, every evidence quote must be found in the OCR text of the stated page, after normalising whitespace and case. A value with no matching evidence is treated as possibly invented. Second, the response's stop reason must be a normal finish, not a token limit. A cut-off response that happens to parse is still incomplete.

import re

def norm(s: str) -> str:
    return re.sub(r"\s+", " ", s).strip().lower()

def unsupported_fields(inv: Invoice, ocr_pages: dict[int, str]) -> list[str]:
    missing = []
    for ev in inv.evidence:
        page_text = norm(ocr_pages.get(ev.page, ""))
        if norm(ev.quote) not in page_text:
            missing.append(ev.field)
    return missing

Two schema habits help as much as validation. Make absence explicit: a nullable po_number with “null if not printed” in the description gets far fewer invented values than a required string. And ask for counts the model can see, like printed_line_count, so you can detect when the list it returned is shorter than the document.

Confidence and human review queues

IDP services return confidence scores. Textract, for example, gives a 0 to 100 score per item and suggests rules such as flagging anything below a threshold.² LLMs do not give you a calibrated per-field number. Asking the model to rate its own confidence produces a field that looks useful and is not. So we build confidence from signals we can check:

The review screen matters as much as the routing. Show the page image with the source of each field highlighted, the extracted value, and the specific reason the document was routed. Reviewers who see “lines do not sum to subtotal” fix the problem in seconds. Reviewers who see a full form and no reason re-key the whole document, and the automation gains disappear.

Cross-checks against systems of record

The strongest check on an extracted invoice is not another model. It is the data you already hold. In accounts payable, these checks catch both extraction errors and real-world problems:

• Supplier match. Resolve the supplier against the vendor master by tax ID first, then name. An unknown supplier is never posted automatically.
• PO matching. Two-way match (invoice against PO price and quantity) or three-way match (plus goods receipt), with tolerances agreed with finance.
• Totals and tax. Recompute tax from the lines and the supplier's expected rate, not just the printed figure.
• Duplicates. Same supplier, invoice number and amount, and also near-duplicates: the same amount and date with an invoice number that differs only by punctuation or a leading zero.
• Bank details. If the bank account on the invoice differs from the vendor master, it always goes to a human and is verified out of band. This is a fraud control as much as a data quality one.

Evaluating extraction

Measure at field level and at document level, because they tell different stories. Here is a worked example with labelled assumptions, not client data.

Assumptions: 200 labelled invoices, 12 header fields each, so 2,400 field slots. In the ground truth, 2,300 fields are present and 100 are legitimately absent (no PO number printed, for example). The pipeline returns 2,290 values:

• Precision = correct / returned = 2,231 / 2,290 = 97.4%.
• Recall = correct / present = 2,231 / 2,300 = 97.0%.

That sounds excellent. Now look at documents. Suppose the 84 field errors fall across 52 invoices, so 148 of 200 are fully correct (74%). The routing rules send 61 invoices to review, and 48 of them contain an error. The other 139 go straight through, and 4 of those still contain an error.

• Straight-through rate = 139 / 200 = 69.5%.
• Escaped error rate = 4 / 139 = 2.9% of auto-posted invoices.
• Review precision = 48 / 61 = 79% of reviewed invoices actually needed a fix.

Field accuracy of 97% turned into 70% automation with 4 bad postings per 139. Those are the numbers finance cares about. Track them per field (totals and bank details matter more than a supplier's phone number), per supplier and per document type, and look hard at the escaped errors: each one is a missing check. Invented values deserve their own line in the report, because they are the errors reviewers are least likely to catch.

Cost per page

List prices as of September 2026. The AWS examples use US West (Oregon); Azure figures are pay-as-you-go for East US. All are for the first volume tier.

LLM cost depends on tokens, so it needs assumptions. We assume 3,000 input tokens per page (page image, extracted text, instructions and schema) and 500 output tokens. Token counts per page vary a lot by provider and content, as the figures above show, so measure yours with the provider's token counter.

• Low-cost tier, at $0.30 per million input tokens and $2.50 per million output (Gemini 2.5 Flash's list price):¹⁴ 3,000 × $0.30/M + 500 × $2.50/M = $0.0009 + $0.00125 = about $2.15 per 1,000 pages.
• Mid tier, at $3 and $15 per million (Claude Sonnet 4.5's list price):¹⁵ $0.009 + $0.0075 = about $16.50 per 1,000 pages.

Both providers offer batch APIs at roughly half price, which suits document work that does not need an answer in seconds.¹⁵¹⁴

Now a monthly example. Assume 10,000 invoices of two pages each (20,000 pages), and a fully loaded reviewer cost of $40 an hour at three minutes per reviewed invoice, or $2 per review:

The extraction line is noise next to the review line. Halving the review rate saves $3,000 a month in this example; switching to the cheapest extractor saves at most $157. Spend on whatever lowers the review rate without raising escaped errors: better checks, a second extractor on hard documents, cleaner review tooling.

Privacy and data residency

Invoices, claims and contracts contain personal and commercial data, so check each service's data terms before the pilot, not after:

• Training and improvement. Amazon Textract may store and use inputs to improve AWS machine-learning services unless you opt out with an AWS Organizations AI services opt-out policy.² Google does not use prompts or responses from paid Gemini API services to improve its products, but unpaid use is different: Google may use that data, human reviewers may read it, and Google says not to submit sensitive or personal information.¹⁶ Make sure nobody prototypes with real documents on a free tier.
• Retention. Azure Document Intelligence processes data in the resource's region and keeps inputs and results for 24 hours after an analysis completes, unless you delete them earlier through the API.¹⁷
• Region. Pin processing to the region your data rules require, for both the IDP service and the model endpoint. Generative features are often available in fewer regions than classic OCR; Google lists a short set of regions for generative Custom Extractor, for example.³
• Your own logs. The pipeline's traces, review queue and evaluation set hold the same sensitive data. Give them the same retention and access rules as the source documents.

Failure modes

Checklist

1. Sort your documents by the traits that make them hard, and sample each group for evaluation.
2. Label at least a couple of hundred real documents, including the ugly ones, before choosing a vendor.
3. Pick the approach per document group: IDP service, LLM, or hybrid.
4. Use schema-constrained output, with nullable fields for anything that may be absent.
5. Validate types, arithmetic, completeness and stop reason in your own code.
6. Require evidence quotes for key fields and match them against OCR text.
7. Cross-check against the vendor master, purchase orders, receipts and past invoices.
8. Route to review with a specific reason, and show reviewers the source on the page.
9. Report precision, recall, straight-through rate and escaped error rate, per field and per supplier.
10. Budget for review time, not just per-page prices.
11. Confirm data use, retention and region for every service before real documents go in.
12. Turn every reviewer correction into a new evaluation example.

If you want this pipeline running on your AP inbox without building it yourself, our Documents & AP agent is set up in your own tools and run for you.

Sources

1. Ouyang et al., “OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations,” CVPR 2025, arXiv:2412.07626 — arxiv.org/abs/2412.07626
2. AWS, Amazon Textract FAQs — aws.amazon.com/textract/faqs/
3. Google Cloud, Document AI documentation — “Processor list” — docs.cloud.google.com/document-ai/docs/processors-list
4. Microsoft Learn, “What is Azure Document Intelligence in Foundry Tools?” — learn.microsoft.com/en-us/azure/ai-services/document-intelligence/overview
5. Microsoft Learn, “Document layout analysis” (Document Intelligence layout model) — learn.microsoft.com/en-us/azure/ai-services/document-intelligence/prebuilt/layout
6. Anthropic, Claude API documentation, “PDF support” — platform.claude.com/docs/en/build-with-claude/pdf-support
7. Google, Gemini API documentation, “Document understanding” — ai.google.dev/gemini-api/docs/document-processing
8. OpenAI API documentation, “File inputs” (PDF files) — developers.openai.com/api/docs/guides/pdf-files
9. OpenAI API documentation, “Structured model outputs” — developers.openai.com/api/docs/guides/structured-outputs
10. Anthropic, Claude API documentation, “Structured outputs” — platform.claude.com/docs/en/build-with-claude/structured-outputs
11. AWS, Amazon Textract pricing — aws.amazon.com/textract/pricing/
12. Google Cloud, Document AI pricing — cloud.google.com/document-ai/pricing
13. Microsoft Azure, Azure Document Intelligence pricing (pay-as-you-go S0 rates for East US, read from the Azure Retail Prices API) — azure.microsoft.com/en-us/pricing/details/ai-document-intelligence/
14. Google, Gemini Developer API pricing — ai.google.dev/gemini-api/docs/pricing
15. Anthropic, Claude API documentation, “Pricing” — platform.claude.com/docs/en/about-claude/pricing
16. Google, Gemini API Additional Terms of Service — ai.google.dev/gemini-api/terms
17. Microsoft Learn, “Data, privacy, and security for Document Intelligence” — learn.microsoft.com/en-us/azure/foundry/responsible-ai/document-intelligence/data-privacy-security

Prices, limits and product names come from each vendor's documentation and price list. They change often, so confirm against the linked page before budgeting.