Passport OCR with AI: Combining MRZ Parsing and Claude Vision
Neither classic OCR nor an AI vision model alone could read WhatsApp passport photos reliably enough for a visa application. Running both and merging them field by field could.
By Wasif Rehman · · AI document processing
In short
- MRZ parsing is trustworthy when its check digits validate, but it fails too often on phone photos to stand alone.
- A vision model covers messy photos and extra fields, but it can return a plausible wrong value instead of failing.
- Merge field by field: verified MRZ data wins, vision fills the gaps, and disagreements are flagged for a person.
- Put two vision providers behind one interface so bursts of work never stall on one outage.
Why passport OCR from WhatsApp photos is hard
For UmrahFlow, my Umrah visa-automation platform, I needed to turn WhatsApp photos of passports into structured data: names, passport numbers, dates and nationality. It had to be reliable enough to feed a government visa submission.
The photos are taken in queues and living rooms, and they arrive with every problem a camera can add:
- Glare across the laminate
- Fingers over the corners
- Perspective skew from holding the phone at an angle
- Compression from WhatsApp itself
Neither reader on its own was good enough. Running both and merging them turned out to be the most interesting engineering problem in the whole project.
MRZ parsing: trustworthy, but brittle on phone photos
Every passport has an MRZ, the machine-readable zone: two lines of <<<-padded text at the bottom of the photo page. It is designed for machines, with fixed positions, a restricted character set, and check digits over the passport number, the birth date and the expiry date.
The problem is the input. On real-world phone photos, a local MRZ pass fails or garbles far too often to run a business on.
Claude vision OCR: reads anything, including what is not there
Claude's vision reads passports impressively: skewed, glared, even partly covered. It also reads the visual zone, the printed fields, which holds data the MRZ doesn't carry, such as place of birth.
But a vision model fails in the dangerous way. It does not say "unreadable". It produces a plausible passport number.
How the two readers are merged, field by field
The OCR service runs both passes and merges them field by field. The rules are simple and boring on purpose:
| Situation | Result | Why |
|---|---|---|
| MRZ parses and its check digits validate | MRZ wins for the fields it covers | It verifies itself and cannot hallucinate |
| The field is not in the MRZ (for example, place of birth) | Claude vision fills it | Only the visual zone has it |
| The MRZ fails entirely on this photo | Claude vision covers the photo | A reading is better than none, and a person reviews it |
| Both read the field and disagree | Neither wins; the field is flagged low confidence | The reviewer should look exactly where the readers diverge |
Downstream, a person sees the extracted fields next to the original photo. Disagreements are kept as flags instead of being resolved silently, so their attention goes where it is needed.
A fallback chain for the vision pass
The vision pass has its own fallback: Claude first, OpenAI if Claude errors or times out.
- WhatsApp photo
- Local MRZ pass
- Claude vision (OpenAI fallback)
- Field-by-field merge
- Human review
Provider redundancy matters more here than in most systems. Input arrives in bursts, such as a group's ten passports right before a deadline, and the whole agency's intake stalls if extraction is down. Both providers sit behind one internal interface, so the merge logic neither knows nor cares which model did the reading.
Building the OCR API to its consumer's contract
One unglamorous detail did a lot of work. The service returns fields in the exact shape the existing dashboard's upload code expects: snake_case fields nested under a primary key. The dashboard predates my services and I had committed to not changing its code, so the OCR API was built to its consumer's expectations rather than my own preferences.
| Endpoint | Purpose |
|---|---|
POST /ocr | Multipart photo uploads |
POST /ocr/url | Images that are already stored |
POST /pdf | Scanned documents |
GET /healthz | Health check used by the end-to-end smoke test |
More on building services to an existing contract in Building Microservices to Someone Else's Contract.
A pattern you can reuse for any AI document reader
The approach applies to any "AI reads a document" problem:
- Pair a deterministic validator Check digits, checksums, or a regex with a known structure.
- With a flexible reader A vision model that copes with messy real-world input.
- Let the validator win where it can Trust verified fields over plausible ones.
- Surface disagreement to a person Never resolve it silently.
You get the vision model's coverage without inheriting its confidence problem.
Frequently asked questions
What is MRZ parsing?
The MRZ is the two lines of machine-readable text at the bottom of a passport photo page. Parsing it gives fixed-position fields plus check digits for the passport number, birth date and expiry date, so a valid read can be trusted almost completely.
Can an AI vision model read passports accurately?
Claude's vision reads skewed, glared and partly covered passports well, including the printed visual zone. The catch is that when it is unsure it can still return a plausible value, so it needs a validator or a human check beside it.
How do you stop AI OCR from hallucinating values?
Let a deterministic validator such as MRZ check digits win wherever it can, use the vision model only to fill gaps, and flag any field where the two readers disagree for a person to review.
Why keep a person in the loop for visa data?
A wrong passport number in a visa application does real harm and looks correct. In UmrahFlow, staff review extracted fields next to the original photo before anything is queued for submission.
Case study: UmrahFlow