Case Study: Cutting Manual Document Processing by 80% for a Logistics Operator

A mid-market logistics operator was processing 12,000+ bills of lading, customs forms, and proof-of-delivery scans per month — almost entirely by hand. The operational data was trapped in PDFs, manual entry errors created billing disputes, and the team could not staff up fast enough to match growth.

This case study walks through the architecture, evaluation strategy, and rollout plan behind the production system we shipped.

The Problem

• 12,000+ documents per month across five formats, three languages
• Manual data entry averaged 6 minutes per document, with a 4% error rate
• Errors surfaced downstream as billing disputes that cost ~$180k per year
• Operational reporting lagged by 7–10 days because the data did not exist in structured form until weeks after the shipment

The team had previously tried an off-the-shelf OCR product. It hit ~70% field accuracy and required manual review on every document, which provided no real lift over typing.

Why the Previous Attempts Failed

Two patterns. The first was a pure OCR vendor — accurate on text but unable to understand structure (tables split across pages, multi-line addresses, line items with merged cells). The second was a pure LLM approach — ChatGPT-style prompting on scanned images. It hallucinated quantities on poor scans and produced non-reproducible outputs.

Neither approach separated extraction from interpretation. Both asked one component to do too much.

The Architecture

The system is a five-stage pipeline. The LLM only enters at stage four, and only for narrow tasks (resolving ambiguous addresses, normalizing carrier names, inferring missing field types from context). Everything else is deterministic.

What Each Layer Does

Classifier routes each document to the correct downstream schema. Trained on 2,000 labeled examples; runs in 40ms per page.

Layout-aware OCR uses a structure-preserving parser (not flat text OCR). Tables stay tables. Multi-page line items reassemble correctly. Confidence scores attach to every extracted token.

Structural decomposition applies deterministic rules: line items reconcile against totals, address blocks are parsed into components, weight/volume units are normalized.

Controlled LLM handles the 15% of fields where deterministic rules cannot decide. The model receives structured candidates, never raw scans. Outputs are schema-constrained.

Schema validation is the gate before ERP sync. Totals must reconcile to the cent. IDs must match expected formats. Failed validation routes to a human review queue with the specific failure highlighted.

Evaluation Strategy

A 400-document golden set covering all five formats, both clean and degraded scans. Tracked separately:

• Field-level accuracy (per-field, not aggregate)
• Document-level pass rate (full document, no manual touch required)
• Confidence calibration (does the system know when it does not know)

Every model or rule change ran against the golden set in CI before deployment.

Rollout

Shadow mode for six weeks — system ran in parallel with manual entry, outputs compared but not used. Then a gated rollout: easy document types first (PoDs), then mid-complexity (BoLs), then customs. Human review queue was the safety net throughout.

Outcomes

• 80% reduction in manual processing time across all document types
• Document-level pass rate of 93% — only 7% routed to human review
• Field-level accuracy of 99.2% on validated outputs
• Billing disputes down ~70% within the first quarter
• Operational data available within minutes of document receipt, instead of days

Tradeoffs Worth Naming

• The deterministic layers are domain-specific and took three engineer-weeks each to build. They will not transfer to a different document type without work.
• Human review remains essential for low-confidence cases. The system makes the humans faster; it does not eliminate them.
• The eval set has to be maintained as document formats drift. Budget for it.

Key Takeaways

The lift did not come from a better model. It came from an architecture that kept the model on a short leash and let deterministic components do the work they were better suited for. That same pattern applies to almost every document intelligence problem worth solving.