OCR in 2026: document understanding beyond plain text extraction

OCR used to mean one thing: turn pixels into characters.

That definition is no longer enough.

A useful OCR pipeline in 2026 often has to answer much richer questions. Which text is a heading? Which paragraph comes next in reading order? Is a sequence of numbers a table row or merely several unrelated text fragments aligned on the page? Which value belongs to which field? Where did an extracted fact come from? Should a PDF page even be sent through OCR, or does it already contain a reliable machine-readable text layer?

Modern document understanding treats text recognition as one stage in a larger pipeline. Current tools reflect that shift. As of August 12, 2026, Tesseract 5.5.3 is the current upstream release; PyMuPDF 1.28.2 is current on PyPI; Docling is at 2.119.0; and PaddleOCR is at 3.7.0, with PaddleOCR-VL-1.6 representing its current vision-language document-parsing line.

This tutorial builds the mental model first, then implements a runnable hybrid PDF parser that preserves geometry instead of throwing it away.

The critical distinction: OCR versus document understanding

Imagine scanning this small inventory table:

Item          Qty    Unit price
Keyboard       12       79.00
Mouse          30       24.00
USB hub         8       39.00

A text-recognition engine might return:

Item Qty Unit price Keyboard 12 79.00 Mouse 30 24.00 USB hub 8 39.00

Every visible character might be recognized correctly, yet most of the useful information has disappeared.

We have lost:

• row boundaries;

• column boundaries;

• header relationships;

• the fact that 79.00 describes the keyboard rather than the mouse;

• coordinates that could highlight the value on the original page;

• confidence or quality information;

• the hierarchy of the surrounding document.

Document understanding tries to preserve those relationships.

A more useful representation looks conceptually like this:

{
  "type": "table",
  "headers": ["Item", "Qty", "Unit price"],
  "rows": [
    ["Keyboard", "12", "79.00"],
    ["Mouse", "30", "24.00"],
    ["USB hub", "8", "39.00"]
  ],
  "page": 1,
  "bbox": [0.08, 0.21, 0.78, 0.43]
}

That distinction matters for search, retrieval-augmented generation, accessibility, analytics, workflow automation, and any application where downstream software must reason about a document rather than merely display its words.

A document AI system therefore needs more than character recognition. It needs at least some combination of geometry, layout, reading order, structure, provenance, and semantic validation.

Think in layers, not in one OCR call

A production document pipeline becomes easier to design when you separate the jobs it is performing.

1. Input classification

First determine what you actually received.

A PDF might be:

• born digital, with embedded machine-readable text;

• a pure scan containing only page images;

• hybrid, with native text on some pages and scanned material on others;

• equipped with a broken or misleading text layer;

• multi-column;

• table-heavy;

• full of figures, headers, footers, annotations, or rotated pages.

Sending every PDF through OCR wastes computation and can reduce extraction quality. PyMuPDF can extract native words directly with Page.get_text("words"), while Page.get_pixmap() can rasterize a page when OCR really is needed. Its current documentation explicitly recommends OCR as a fallback when normal text extraction is empty or implausible.

2. Rasterization

When a page does need OCR, the recognizer ultimately needs image data.

Important variables include:

• resolution;

• page rotation;

• skew;

• perspective distortion;

• text size;

• compression artifacts;

• shadows;

• contrast.

PyMuPDF’s current Page.get_pixmap() API accepts a dpi argument directly. Its documentation uses 300 DPI as an explicit example for higher-resolution rendering.

That does not make 300 DPI a universal optimum. It is simply a useful baseline for printed documents before you start tuning.

3. Text recognition

Traditional OCR detects and recognizes text.

Tesseract remains useful because it can operate locally and emit much more than a plain string. Its current command-line interface supports TSV output containing hierarchy identifiers, bounding boxes, confidence values, and recognized words.

That richer output is exactly what we will use.

4. Layout analysis

Layout analysis tries to identify regions such as:

• titles;

• section headings;

• body paragraphs;

• lists;

• tables;

• captions;

• figures;

• formulas;

• headers and footers.

This is where OCR becomes document understanding.

Current Docling documentation describes advanced PDF processing that includes page layout, reading order, table structure, formulas, code, OCR, and a unified structured document representation.

5. Reading order

Consider a two-column article.

A naive coordinate sort can produce:

left line 1
right line 1
left line 2
right line 2
left line 3
right line 3

The intended sequence may actually be the complete left column followed by the complete right column.

Reading order is therefore not cosmetic. It changes meaning.

Even a perfect transcription can become unusable if the sequence of elements is wrong.

6. Structural reconstruction

Tables are the clearest example.

A useful table parser must infer:

• rows;

• columns;

• blank cells;

• row spans;

• column spans;

• headers;

• nested headers;

• which text belongs to which cell.

These relationships are often expressed spatially rather than through characters.

7. Semantic extraction

Only after structure is reasonably trustworthy should downstream logic ask questions such as:

• What is the total?

• Which date is the delivery date?

• Which quantity corresponds to this SKU?

• Which heading governs this paragraph?

• Which server recorded the largest latency?

A language model can help here, but feeding it scrambled text forces it to repair document structure before it can even start the actual task.

8. Provenance

A serious extraction pipeline should be able to answer:

Where did this value come from?

Useful provenance includes:

• source filename;

• page number;

• bounding box;

• extraction route;

• recognition confidence where available;

• structural parent, such as a table cell or paragraph.

That information makes review, debugging, highlighting, and citations dramatically easier.

The 2026 tool landscape

There is no single correct stack.

The useful question is not “Which OCR library wins?” but “Which stage of my pipeline am I solving?”

Tesseract 5.5.3

Tesseract 5.5.3 was released on July 24, 2026 and is the current upstream release. It remains a strong local recognition engine when you need text plus positional formats such as TSV or hOCR.

It is not, by itself, a complete semantic document parser.

PyMuPDF 1.28.2

PyMuPDF 1.28.2 was released on August 6, 2026. It provides low-level PDF text extraction, geometry, rendering, and document manipulation.

For hybrid OCR pipelines, it is especially useful as the routing and rasterization layer.

Docling 2.119.0

Docling 2.119.0 was released on August 10, 2026. Its DocumentConverter remains the main document-conversion entry point, while the resulting DoclingDocument supports structured traversal and exports including Markdown and dictionaries suitable for JSON serialization.

Docling sits considerably higher in the abstraction stack than a raw OCR engine.

PaddleOCR 3.7.0 and PaddleOCR-VL-1.6

PaddleOCR 3.7.0 was released on June 11, 2026. The same current project line includes PaddleOCR-VL-1.6, released on May 28, 2026 for document parsing involving text, tables, formulas, charts, and other visually structured content.

The engineering lesson is more important than the brands:

choose tools by pipeline stage.

Build a hybrid parser that keeps geometry

We will now build a small PDF parser with this policy:

1. Open a PDF.

2. Extract native words from each page.

3. If enough native words exist, preserve them.

4. Otherwise rasterize the page at 300 DPI.

5. Call Tesseract directly.

6. Parse Tesseract TSV output.

7. Normalize every bounding box into the range 0 to 1.

8. Reconstruct simple line groups.

9. Save the result as JSON.

The runnable path deliberately does not require a Python wrapper around Tesseract. Python invokes the current Tesseract executable through subprocess and parses its documented TSV format directly.

That keeps the Python dependency set small and current.

Create the environment

Use Python 3.10 or newer. Docling 2.119.0 itself requires Python 3.10 or newer, so that is a useful shared baseline if you intend to follow the higher-level example later.

Create a project directory:

ocr-course/
├── parse_document.py
├── requirements.txt
└── sample.pdf

Create the virtual environment:

python -m venv .venv

Activate it on Linux or macOS:

source .venv/bin/activate

Put the current PyMuPDF version into requirements.txt:

PyMuPDF==1.28.2

Install exactly that environment:

python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Pinning matters in a course example. An unbounded upgrade command may silently install a different release months later, changing behavior while the article remains unchanged.

PyMuPDF 1.28.2 is the current release as of August 12, 2026.

Install and verify Tesseract

On Debian or Ubuntu systems, the distribution package can be installed with:

sudo apt-get update
sudo apt-get install -y tesseract-ocr

The important caveat is that Linux distribution repositories do not guarantee the same upstream Tesseract version. A stable distribution can lag behind the latest Tesseract release.

Verify what you actually received:

tesseract --version | head -n 1

For the environment described in this article, the expected first line is:

tesseract 5.5.3

If your distribution repository returns an older release, follow the upstream Tesseract installation guidance for your platform rather than assuming the package-manager version equals the current upstream version. The current upstream release is 5.5.3.

On macOS with Homebrew:

brew install tesseract
tesseract --version | head -n 1

Again, verify the version instead of assuming what the package manager installed.

The complete parser

Create parse_document.py:

import argparse
import csv
import io
import json
import shutil
import subprocess
import tempfile
from collections import defaultdict
from pathlib import Path

import pymupdf


OCR_DPI = 300
MIN_NATIVE_WORDS = 20
TESSERACT_BIN = "tesseract"


def normalize_bbox(x0, y0, x1, y1, width, height):
    return [
        round(x0 / width, 6),
        round(y0 / height, 6),
        round(x1 / width, 6),
        round(y1 / height, 6),
    ]


def ensure_tesseract():
    executable = shutil.which(TESSERACT_BIN)

    if executable is None:
        raise RuntimeError(
            "Tesseract was not found on PATH. "
            "Install Tesseract and verify it with 'tesseract --version'."
        )

    return executable


def build_lines(tokens):
    groups = defaultdict(list)

    for token in tokens:
        groups[token["line_id"]].append(token)

    lines = []

    for line_id, words in groups.items():
        words.sort(key=lambda item: item["bbox"][0])

        x0 = min(word["bbox"][0] for word in words)
        y0 = min(word["bbox"][1] for word in words)
        x1 = max(word["bbox"][2] for word in words)
        y1 = max(word["bbox"][3] for word in words)

        lines.append(
            {
                "line_id": line_id,
                "text": " ".join(word["text"] for word in words),
                "bbox": [x0, y0, x1, y1],
            }
        )

    lines.sort(key=lambda item: (item["bbox"][1], item["bbox"][0]))

    return lines


def extract_native_page(page):
    page_width = page.rect.width
    page_height = page.rect.height
    words = page.get_text("words", sort=True)

    tokens = []

    for word in words:
        x0, y0, x1, y1, text, block_no, line_no, word_no = word
        text = text.strip()

        if not text:
            continue

        unrotated_rect = pymupdf.Rect(x0, y0, x1, y1)
        rotated_rect = unrotated_rect * page.rotation_matrix

        tokens.append(
            {
                "text": text,
                "confidence": None,
                "bbox": normalize_bbox(
                    rotated_rect.x0,
                    rotated_rect.y0,
                    rotated_rect.x1,
                    rotated_rect.y1,
                    page_width,
                    page_height,
                ),
                "line_id": f"native-{block_no}-{line_no}",
                "word_no": word_no,
            }
        )

    return tokens


def render_page(page, image_path):
    pixmap = page.get_pixmap(
        dpi=OCR_DPI,
        alpha=False,
    )

    pixmap.save(image_path)

    return pixmap.width, pixmap.height


def run_tesseract_tsv(image_path):
    executable = ensure_tesseract()

    command = [
        executable,
        str(image_path),
        "-",
        "--dpi",
        str(OCR_DPI),
        "--psm",
        "3",
        "tsv",
    ]

    completed = subprocess.run(
        command,
        check=True,
        capture_output=True,
        text=True,
        encoding="utf-8",
    )

    return completed.stdout


def extract_ocr_page(page):
    with tempfile.TemporaryDirectory() as temp_dir:
        image_path = Path(temp_dir) / "page.png"
        image_width, image_height = render_page(
            page,
            image_path,
        )

        tsv_text = run_tesseract_tsv(image_path)

    reader = csv.DictReader(
        io.StringIO(tsv_text),
        delimiter="\t",
    )

    tokens = []

    for row in reader:
        if row["level"] != "5":
            continue

        text = row["text"].strip()

        if not text:
            continue

        left = int(row["left"])
        top = int(row["top"])
        width = int(row["width"])
        height = int(row["height"])

        x0 = left
        y0 = top
        x1 = left + width
        y1 = top + height

        confidence = float(row["conf"])

        tokens.append(
            {
                "text": text,
                "confidence": round(confidence, 3),
                "bbox": normalize_bbox(
                    x0,
                    y0,
                    x1,
                    y1,
                    image_width,
                    image_height,
                ),
                "line_id": (
                    "ocr-"
                    f"{row['block_num']}-"
                    f"{row['par_num']}-"
                    f"{row['line_num']}"
                ),
                "word_no": int(row["word_num"]),
            }
        )

    return tokens


def native_text_is_plausible(tokens):
    return len(tokens) >= MIN_NATIVE_WORDS


def parse_pdf(path):
    pages = []

    with pymupdf.open(path) as document:
        for page_number, page in enumerate(
            document,
            start=1,
        ):
            native_tokens = extract_native_page(page)

            if native_text_is_plausible(native_tokens):
                source = "native"
                tokens = native_tokens
            else:
                source = "ocr"
                tokens = extract_ocr_page(page)

            pages.append(
                {
                    "page": page_number,
                    "source": source,
                    "rotation": page.rotation,
                    "tokens": tokens,
                    "lines": build_lines(tokens),
                }
            )

    return {
        "document": str(path),
        "page_count": len(pages),
        "pages": pages,
    }


def main():
    parser = argparse.ArgumentParser(
        description=(
            "Extract native PDF text when available "
            "and fall back to Tesseract OCR."
        )
    )
    parser.add_argument(
        "input",
        type=Path,
    )
    parser.add_argument(
        "--output",
        type=Path,
        default=Path("parsed.json"),
    )

    args = parser.parse_args()

    if not args.input.exists():
        raise FileNotFoundError(args.input)

    if args.input.suffix.lower() != ".pdf":
        raise ValueError(
            "This example expects a PDF input file."
        )

    result = parse_pdf(args.input)

    args.output.write_text(
        json.dumps(
            result,
            indent=2,
            ensure_ascii=False,
        ),
        encoding="utf-8",
    )

    print(f"Wrote {args.output}")


if __name__ == "__main__":
    main()

The parser relies on current documented PyMuPDF APIs for word extraction, page rotation geometry, and rasterization. PyMuPDF documents Page.rotation_matrix as the transform for coordinates in rotated page space, which is why the native-text path applies it before normalizing coordinates.

The OCR path uses Tesseract’s documented TSV output. At word level, that format exposes block_num, par_num, line_num, word_num, coordinates, confidence, and text.

Run the parser

Place a PDF at sample.pdf, then run:

python parse_document.py sample.pdf --output sample.json

For a page containing a recognized word such as Keyboard, the resulting structure will have this general shape:

{
  "document": "sample.pdf",
  "page_count": 1,
  "pages": [
    {
      "page": 1,
      "source": "ocr",
      "rotation": 0,
      "tokens": [
        {
          "text": "Keyboard",
          "confidence": 96.1,
          "bbox": [0.102, 0.233, 0.221, 0.254],
          "line_id": "ocr-1-1-3",
          "word_no": 1
        }
      ]
    }
  ]
}

The exact coordinates and confidence values will, of course, depend on your input document.

The important design choice is that Keyboard is no longer an anonymous substring.

It has:

• a page;

• a normalized bounding box;

• a line relationship;

• an extraction route;

• an OCR confidence value when OCR was used.

That gives later stages something concrete to reason about.

Why call Tesseract directly?

The simplest OCR tutorial often hides everything behind one convenience function that returns a string.

That is not what we want here.

Our code invokes the Tesseract executable and requests TSV:

tesseract page.png - --dpi 300 --psm 3 tsv

Tesseract documents TSV as a first-class command-line output format.

This has several advantages for a course project:

• the OCR engine version is independently verifiable;

• the output format is visible;

• there is no extra Python wrapper between your code and the OCR engine;

• coordinates are preserved;

• word confidence is preserved;

• hierarchical IDs are preserved.

The lesson is broader than Tesseract: do not throw away information at the beginning of the pipeline and then try to reconstruct it later.

Why normalize coordinates?

Native PDF extraction operates in page-space coordinates.

Raster OCR operates in image pixels.

If one page is rendered at 200 DPI and another at 300 DPI, raw pixel coordinates do not share the same scale.

The parser converts boxes into values from 0 to 1.

For example:

[0.10, 0.23, 0.22, 0.25]

means approximately:

• left edge at 10% of page width;

• top edge at 23% of page height;

• right edge at 22%;

• bottom edge at 25%.

Normalized geometry is useful for:

• document viewers;

• bounding-box highlights;

• layout heuristics;

• resolution-independent storage;

• visual grounding;

• provenance;

• downstream models.

In a production system, retaining both the native coordinates and normalized coordinates is often useful. Normalized values are convenient for cross-resolution processing; original values remain valuable when interoperating with PDF-specific APIs.

Rotation is an easy source of invisible bugs

Page geometry deserves special attention.

PyMuPDF exposes both page rotation and transformation matrices, including Page.rotation_matrix for mapping coordinates into rotated page space.

Without handling that transform, you can create a frustrating failure mode:

• OCR boxes look correct;

• native-text boxes look correct numerically;

• but overlays for rotated pages appear in the wrong place.

That is precisely why document parsing needs a coordinate convention.

Pick one orientation for stored geometry and convert every extraction path into it.

The native-text threshold is deliberately a heuristic

Our parser currently decides that native PDF text is usable when:

MIN_NATIVE_WORDS = 20

That is intentionally simple.

It is not a production-quality PDF classifier.

A PDF can contain an invisible or corrupted text layer. It might return hundreds of tokens while still producing:

• garbage Unicode;

• duplicated characters;

• bizarre line ordering;

• text positioned far outside visible areas;

• a text layer unrelated to the visible scan.

Current PyMuPDF guidance explicitly notes that text extraction can be empty or garbled and that detecting scrambled text reliably is difficult.

A stronger routing score might examine:

• extracted character count;

• percentage of printable characters;

• frequency of Unicode replacement characters;

• duplicate text;

• spatial coverage;

• number of page images;

• suspicious coordinates;

• whether words form plausible lines;

• whether expected language patterns occur.

This illustrates an important OCR engineering principle:

confidence belongs to the pipeline, not only to the recognizer.

The cherry on the cake: a real table failure mode documented by a current parser

Here is the table lesson that surprises people even after they have built accurate OCR.

Suppose the source is:

Product      Jan    Feb    Mar
Alpha        100           140
Beta          90    110    130

The February cell for Alpha is intentionally blank.

Now suppose every visible character is recognized perfectly:

Alpha 100 140
Beta 90 110 130

A naive parser might reconstruct:

Product      Jan    Feb    Mar
Alpha        100    140
Beta          90    110    130

Now 140 has moved from March into February.

No character was misrecognized.

The topology failed.

The interesting part is that this is not merely a classroom hypothetical. Current Docling documentation explicitly exposes a table option called do_cell_matching and explains that disabling cell matching can improve output when multiple columns in extracted tables are erroneously merged into one.

That is a concrete, documented example of the separation between OCR correctness and table correctness.

The recognizer can know what the text says while the table parser still misunderstands where that text belongs.

Blank cells are especially dangerous because they encode information through absence plus geometry.

The same principle applies to:

• merged headers;

• row spans;

• column spans;

• borderless tables;

• cells containing multiple lines;

• tables split across pages.

Character accuracy alone can therefore give a dangerously optimistic picture of a document-understanding system.

That distinction is also reflected in current document-parsing benchmarks. PaddleOCR-VL-1.6 reports 96.33% on OmniDocBench v1.6, while its official documentation separately emphasizes improvements across text, formula, and table recognition rather than treating all three as one OCR task.

Upgrade from tokens to structured parsing with Docling

Once you understand what the lower-level pipeline is preserving, a document framework becomes much easier to evaluate.

Install the current Docling release explicitly:

python -m pip install "docling==2.119.0"

Docling 2.119.0 is the current PyPI release as of August 12, 2026.

Create parse_with_docling.py:

import json
from pathlib import Path

from docling.document_converter import DocumentConverter


source = Path("sample.pdf")

converter = DocumentConverter()
result = converter.convert(source)
document = result.document

Path("sample.md").write_text(
    document.export_to_markdown(),
    encoding="utf-8",
)

Path("sample.structured.json").write_text(
    json.dumps(
        document.export_to_dict(),
        indent=2,
        ensure_ascii=False,
    ),
    encoding="utf-8",
)

print("Wrote sample.md and sample.structured.json")

Run it:

python parse_with_docling.py

DocumentConverter remains Docling’s primary conversion entry point, and current examples use result.document, export_to_markdown(), and export_to_dict() in exactly this style.

This is a fundamentally different abstraction level from raw OCR.

Markdown is useful when the next consumer is:

• a human;

• a language model;

• a search indexing pipeline;

• a lightweight publishing workflow.

The structured representation is more useful when code needs:

• element types;

• provenance;

• tables;

• hierarchy;

• deterministic traversal;

• richer validation;

• document-native transformations.

Docling’s internal table model also preserves span metadata such as row spans and column spans even though Markdown itself cannot represent every table structure faithfully.

That is another reason not to confuse a serialization format with the underlying document model.

Configure table structure explicitly

Docling’s current PDF pipeline exposes table processing separately.

A minimal configuration looks like this:

from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import PdfPipelineOptions
from docling.document_converter import (
    DocumentConverter,
    PdfFormatOption,
)


pipeline_options = PdfPipelineOptions(
    do_table_structure=True,
)

converter = DocumentConverter(
    format_options={
        InputFormat.PDF: PdfFormatOption(
            pipeline_options=pipeline_options,
        )
    }
)

result = converter.convert("sample.pdf")

print(result.document.export_to_markdown())

Current Docling pipeline documentation continues to expose do_table_structure and table_structure_options for this purpose.

For the specific documented problem where matching predicted table structure back to PDF cells merges columns incorrectly, you can test:

from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import PdfPipelineOptions
from docling.document_converter import (
    DocumentConverter,
    PdfFormatOption,
)


pipeline_options = PdfPipelineOptions(
    do_table_structure=True,
)

pipeline_options.table_structure_options.do_cell_matching = False

converter = DocumentConverter(
    format_options={
        InputFormat.PDF: PdfFormatOption(
            pipeline_options=pipeline_options,
        )
    }
)

result = converter.convert("sample.pdf")

print(result.document.export_to_markdown())

Current Docling guidance says that disabling cell matching can improve output specifically when multiple columns are erroneously merged.

Do not turn it off blindly.

The right workflow is:

• keep a representative validation set;

• compare configurations;

• measure structural correctness;

• choose the setting that improves your own documents.

A tuning switch is not a universal best practice.

Where PaddleOCR fits

PaddleOCR is worth evaluating when image-first documents dominate your workload.

Examples include:

• scanned forms;

• photographed pages;

• multilingual material;

• complex layouts;

• tables;

• formulas;

• charts;

• seals;

• distorted or skewed documents.

PaddleOCR 3.7.0 is the current package release from June 11, 2026. Its current project documentation describes PP-OCRv6 for text recognition and the PaddleOCR-VL line for richer document parsing.

PaddleOCR-VL-1.6 was released on May 28, 2026. The official project reports a 96.33% score on OmniDocBench v1.6 and highlights text, formula, and table recognition as distinct capabilities.

That does not mean every application should replace a conventional OCR pipeline with a vision-language model.

A smaller deterministic stack can still be preferable when:

• documents have predictable typography;

• CPU-only operation matters;

• reproducibility matters;

• the workload is mostly text recognition;

• infrastructure simplicity is important;

• every intermediate step needs to be easy to audit.

Use a more complicated document model because your documents require it, not because a more complicated model exists.

What about Pillow?

If you later add standalone image preprocessing, Pillow is still a current option. Pillow 12.3.0 was released on July 1, 2026.

You could pin it explicitly:

python -m pip install "Pillow==12.3.0"

The baseline parser above does not need it because PyMuPDF can render each PDF page directly to a temporary PNG.

Fewer dependencies are useful when they also mean fewer moving parts.

Preprocessing: intervene only when evidence says you should

OCR tutorials often evolve into enormous preprocessing pipelines:

• grayscale conversion;

• thresholding;

• sharpening;

• denoising;

• morphology;

• contour detection;

• deskewing;

• contrast manipulation.

Those techniques can help.

They can also destroy information.

Aggressive binarization, for example, can remove:

• thin punctuation;

• decimal points;

• faint table rules;

• lightweight fonts;

• anti-aliased strokes.

Instead of applying ten transformations to every page, make preprocessing measurable.

A better workflow is:

1. render the source;

2. run baseline OCR;

3. record the errors;

4. classify the page failure;

5. apply a targeted transformation;

6. rerun the same evaluation;

7. retain the transformation only if the metric improves.

For a phone photograph, perspective correction may matter more than sharpening.

For tiny print, resolution may matter more than thresholding.

For a borderless table, no amount of sharpening can recover a missing table topology if your system never models table structure in the first place.

OCR confidence is not business truth

Tesseract TSV output provides word-level confidence values.

It is tempting to see:

96

and interpret it as:

There is exactly a 96% probability that this word is correct.

Do not design critical business logic around that interpretation.

Treat recognition confidence as a model-specific quality signal.

It can help you:

• rank uncertain words;

• identify unusually poor pages;

• prioritize human review;

• compare similar outputs from the same pipeline.

But structural and semantic validation can catch errors that recognition confidence cannot.

Suppose a document contains:

Subtotal: 980.00
Tax:       78.40
Total:   1058.40

Your application can validate:

980.00 + 78.40 = 1058.40

If OCR produces:

Total: 105B.40

a business-rule validator can reject the result even if the recognizer itself was relatively confident.

Meaning can validate recognition.

Evaluate structures, not screenshots

A document-understanding system needs a representative test corpus.

Do not evaluate on one perfect invoice or one clean PDF.

Include cases such as:

• born-digital PDFs;

• pure scans;

• mixed native/scanned documents;

• rotated pages;

• multi-column pages;

• faint scans;

• footnotes;

• borderless tables;

• blank cells;

• merged headers;

• tables continuing onto later pages;

• phone photographs;

• skew;

• warping;

• difficult lighting.

Real-world robustness deserves explicit testing. The PaddleOCR-VL-1.5 work, for example, introduced Real5-OmniDocBench specifically around physical distortions including scanning, skew, warping, screen photography, and illumination.

Then choose metrics for the task you actually care about.

For text recognition

Measure:

• character accuracy;

• word accuracy;

• important-field accuracy.

For reading order

Measure:

• whether paragraphs occur in the correct sequence;

• whether columns remain intact;

• whether captions stay near their figures.

For tables

Measure:

• row count;

• column count;

• blank-cell preservation;

• cell assignments;

• merged-cell handling;

• header relationships;

• cross-page continuity.

For application extraction

Measure the final output.

If the business needs the correct product quantity, “the OCR text looked good” is not a useful acceptance criterion.

A system with slightly worse raw character accuracy but correct table cells may be far more valuable than one producing beautiful text with broken structure.

Preserve the raw layer and the interpreted layer

Do not discard the evidence once parsing succeeds.

A robust architecture might retain:

document.pdf
├── original source
├── rendered page images
├── native PDF tokens
├── OCR tokens
├── normalized bounding boxes
├── layout elements
├── tables
├── structured document
└── application-specific fields

Each layer answers a different debugging question.

If an application claims an order quantity is 500, you should ideally be able to trace:

application field
→ table cell
→ recognized token
→ bounding box
→ source page

That trace is enormously useful when something goes wrong.

It also makes human review much easier: instead of showing someone an extracted number in isolation, you can highlight the exact source region that produced it.

Provenance is also valuable for RAG

Retrieval-augmented generation over documents is much more reliable when chunks retain structure.

Compare a chunk containing only:

140

with one carrying:

{
  "text": "140",
  "page": 3,
  "element_type": "table_cell",
  "row_header": "Alpha",
  "column_header": "Mar",
  "bbox": [0.62, 0.31, 0.69, 0.34]
}

The second representation gives a downstream system far more context.

This is why modern document understanding is not merely OCR followed by embeddings.

The quality of the intermediate representation determines how much accidental ambiguity reaches retrieval and generation.

Local processing and sensitive documents

Document-processing systems routinely handle material that should not be uploaded casually:

• contracts;

• internal reports;

• financial documents;

• customer files;

• identity documents;

• proprietary research.

Tool choice therefore has a privacy dimension.

Docling currently advertises local execution and support for air-gapped environments. Its documentation also provides a model-prefetch workflow using docling-tools models download so required model artifacts can be prepared before offline operation.

For any document pipeline, decide explicitly:

• whether page images leave your infrastructure;

• where OCR runs;

• whether model inference is local or remote;

• whether temporary images are retained;

• whether logs contain extracted text;

• how long structured documents are stored;

• whether third-party services receive document content.

“Add document AI” should not silently mean “upload every page somewhere.”

A practical architecture for 2026

For many applications, a sensible architecture looks like this:

Input document
      |
      v
File validation
      |
      v
Page classification
      |
      +--------------------------+
      |                          |
      v                          v
Reliable native text        Scan / image
      |                          |
      |                          v
      |                      OCR engine
      |                          |
      +------------+-------------+
                   |
                   v
             Layout analysis
                   |
                   v
        Table / form structure
                   |
                   v
        Structured document model
                   |
          +--------+---------+
          |                  |
          v                  v
     Search / RAG       Business logic
                              |
                              v
                      Validation / review

The important feature is not any particular model.

It is the routing.

You do not need an expensive multimodal parser to recover a clean paragraph from a text-native PDF when the PDF engine already contains the characters and coordinates.

You probably do need more than raw OCR for a skewed photograph containing multiple columns, a borderless table, blank cells, and merged headers.

The system should know the difference.

What to build first

If you are starting an OCR project in 2026, resist beginning with the largest possible model.

Start by making the output observable.

Build these pieces first:

• native PDF extraction;

• raster fallback;

• word bounding boxes;

• page numbers;

• extraction-source metadata;

• recognition confidence;

• normalized geometry;

• structured JSON;

• a small evaluation corpus.

Then inspect where the baseline fails.

If failures mostly involve reading order, add layout understanding.

If failures mostly involve tables, add table-structure recognition.

If scans involve complicated physical distortion, multilingual content, formulas, charts, or visual elements, evaluate a current vision-language document parser.

If native text is already excellent, do not OCR it again simply because your product is called an OCR system.

The mental model to keep

The most important change in OCR is not that recognizers have become more sophisticated.

It is that the output contract has changed.

Plain text asks:

What characters appeared on the page?

Document understanding asks:

What information appeared, how was it organized, where did it come from, and how confidently can downstream software use it?

Those are different engineering problems.

The hybrid parser in this tutorial gives you a deliberately transparent baseline. It shows which pages were routed through native extraction, which pages required OCR, what geometry was recovered, and how the words relate to lines.

A structured framework such as Docling can then add layout, reading order, tables, hierarchy, and richer serialization. A current image-first stack such as PaddleOCR can address substantially harder visual parsing cases.

The next time you evaluate an OCR system, do not stop at:

Did it read the words?

Ask:

Did it preserve enough document structure that software can safely use what it read?

Take one real PDF from your own workflow today. Run the hybrid parser, inspect the native-versus-OCR routing and bounding boxes, then process the same file with Docling. Compare reading order, blank table cells, merged headers, page provenance, and structured output. That side-by-side test is the fastest way to see where plain OCR ends and modern document understanding begins.