Same document, two architectures. Only one is reproducible.
Why LLM-Only Document Extraction Fails — And the Deterministic-First Fix
Most engineering teams approach document extraction the same way: connect an LLM to a PDF parser, write a prompt, ship a demo. It works on the first ten documents. It fails on the next ten thousand.
The failure is not the model. It is the architecture around it.
Production document intelligence is not a prompting problem. It is a systems problem.
The Problem: Documents Are Not Text
Engineering documents — HVAC blueprints, electrical schematics, mechanical drawings, CAD exports — are geometric and relational systems. They encode meaning through position, scale, layering, and symbolic convention. Flattening them to text or images for an LLM throws that meaning away.
What the model receives is a lossy projection of the real document. What we then ask it to do is reconstruct, by inference, the structure that was deleted before it ever saw the file. That is not extraction. That is guessing.
Why LLM-Only Pipelines Fail
The failure modes are not random. They are consequences of asking a probabilistic language model to operate on structurally precise engineering data.
These are not prompt-engineering bugs. They are inherent to a single-stage pipeline where the LLM is asked to be parser, classifier, validator, and structurer at once.
The Deterministic-First Architecture
Invert the pipeline. Instead of letting a language model interpret raw geometry, extract structure deterministically first and let the model reason over a clean, validated representation.
1. Deterministic Extraction
No interpretation. Faithful capture of what the document contains: PDF structure parsing (vector vs raster), CAD extraction (layers, blocks, entity tables), targeted OCR on regions that genuinely need it. The output is raw but complete and lossless.
2. Structural Decomposition
Group raw fragments into meaningful units using deterministic rules: zones, system-level segmentation (HVAC networks, electrical circuits), repeated component detection via geometric similarity, adjacency and connectivity graphs. The document becomes a structured system model.
3. Schema-Based Representation
Entities are formalized against strict schemas, every field validated and traceable to the source geometry that produced it. Ducts become typed flow paths. Equipment becomes labeled system nodes. Dimensions become unit-checked measurements. Provenance is built in.
4. Controlled LLM Integration
Only now does the model enter the pipeline. It never sees a raw PDF. It operates on structured data, scoped to narrow tasks: resolving ambiguous labels, classifying from constrained vocabularies, handling edge cases the rule layers explicitly defer.
5. Production Outputs
Artifacts downstream workflows actually consume: CSV/Excel for estimation, JSON for APIs and databases, BIM-compatible structured datasets. Every output carries provenance — source page, source geometry, extraction stage, validation status.
Tradeoffs
- Engineering effort up front. Deterministic extractors take longer to build than a single prompt. They also keep working.
- Domain coupling. Rule layers are domain-specific; the LLM layer is reusable. Plan for that split.
- Coverage gaps. Some entities will always fall through to the model. Make sure the model knows when it is guessing.
Expected Outcomes
Reproducible outputs across runs. Auditable provenance for every value. Confidence calibration that survives volume. And — most importantly — a system whose failures are diagnosable instead of mysterious.
Key Takeaways
LLM-only extraction fails because the model is asked to do work that should never have reached it. The discipline is to reduce ambiguity before the model is involved, treat the LLM as a narrow reasoning layer, and make the rest of the system deterministic, observable, and maintainable.