On May 11, 2026, InfoQ released Local-First AI Inference: A Cloud Architecture Pattern for Cost-Effective Document Processing, sharing an architectural pattern that demonstrated a 75% API cost reduction plus faster processing times when handling 4,700 engineering drawing PDFs.
Based on the simple yet powerful idea of "processing 70% to 80% of documents with local deterministic extraction and sending only the remaining 20% to 30% of edge cases to cloud LLMs," this has the potential to become the standard pattern for document DX custom projects struggling with cost explosions. It offers an optimization approach across both "accuracy" and "cost," complementing 99.94% Recognition Accuracy with OCR × LLM Hybrids.
Why calling LLMs for every request explodes costs
| Item | All-LLM | Local-First |
|---|---|---|
| Monthly document volume | 1,000,000 | 1,000,000 |
| LLM calls | 1,000,000 | 200,000 |
| API cost | 1,000,000 yen | 250,000 yen |
| Processing time | 5 seconds / document on average | 1.5 seconds / document on average |
| Offline resilience | Zero (halts on API outages) | 70% to 80% remains operational |
This follows the same cost-explosion structure addressed in 45x Cost in Computer Use Client Development, requiring you to design not "whether to use an LLM or not," but "which specific documents should use an LLM."
4-stage routing in Local-First AI inference
Stage 1: Deterministic extraction (fast and low cost)
Using regular expressions, templates, and rule-based methods, process "documents with clearly structured formats." For documents with fixed formats—such as invoices, contract templates, and application forms—most are completed right here.
Stage 2: Lightweight model extraction (local inference)
Run lightweight models such as LayoutLM, Donut, or Qwen-VL-Local on local GPUs or CPUs to process documents with less distinct structures. This completes at under 1 yen per item.
Stage 3: Cloud LLM calls (high cost, high accuracy)
Send documents to Claude, GPT-5.5, or Gemini 2.5 only when "both deterministic extraction and lightweight models failed." Limiting this to 20% to 30% of total volume dramatically compresses API costs.
Stage 4: Human review queue
Documents where "the LLM had low confidence" are forwarded to the human review UI. A realistic proportion is 1% to 5% of the total.
Three implementation phases built in custom development
Phase 1: Routing rule design (4 weeks)
Analyze document samples from the past month and measure "what percentage can pass through deterministic extraction." Align on the rationale for routing ratios with the client.
Phase 2: Pipeline construction (8–10 weeks)
Build a standard architecture of storage (S3 / GCS) + queues (SQS / Pub/Sub) + local inference workers + cloud LLM workers + review UI. Design it with Auto Scaling so that traffic routes to cloud services only during peaks.
Phase 3: Continuous optimization (monthly operations)
We run a continuous improvement loop that re-evaluates routing ratios on a monthly basis and incorporates into Stage 1 and Stage 2 the "patterns of documents currently sent to LLMs." This is the document-processing edition of continuous improvement covered in DORA / SPACE / Core 4 ROI Client Projects.
Standard technology stack set for custom development
| Layer | Recommended technology | Alternative |
|---|---|---|
| Deterministic extraction | unstructured + regex | In-house parser |
| Lightweight models | LayoutLM / Donut | Qwen-VL-Local |
| GPU environment | g5.xlarge / GCP L4 | On-premises GPU |
| Cloud LLMs | Claude Sonnet / GPT-5.5 | Gemini 2.5 |
| Queue | SQS / Pub/Sub | Cloud Tasks |
| Observability | OpenTelemetry + Grafana | Datadog |
| Cost management | Cost Explorer + custom dashboard | CloudHealth |
In particular, by "tracking routing ratios on a daily dashboard," you can run the loop of "cost anomalies → redesigning routing rules" within a matter of hours.
Which projects Local-First fits best
| Project type | Impact of Local-First |
|---|---|
| Processing 100,000+ invoices monthly | Reduces monthly API costs by 300,000 to 700,000 yen |
| Identity verification documents in finance | Keeps data confined within VPCs |
| Engineering drawing processing in manufacturing | Makes bulk processing of large page volumes viable |
| Public sector application intake | Accommodates peak-time scaling |
| Annotation of medical charts | Designed not to expose PHI to LLMs |
Five clauses to include in custom development contracts
| Clause | Details | What the client should verify |
|---|---|---|
| Routing ratio SLA | Target ratio to process in Stage 1 / Stage 2 | Tolerance for divergence from measured values |
| API cost ceiling | Automated shutoff via monthly budgets | Demarcation of responsibility for business impact |
| Data egress scope | Document types sent to cloud LLMs | Consistency with confidentiality classifications |
| Tracking model updates | Re-evaluation frequency for lightweight models | Responsibility for evaluation effort |
| Continuous improvement scope | Work scope for monthly optimization | Risk when improvement halts |
Four common pitfalls
Pitfall 1: Estimating based on an unquestioned "LLM-only is fine" request
Clients often want "LLM-only initially," but cases where projects shut down due to cost explosions after three months are frequent. Proposing Local-First from the beginning serves the client's interests over the long term.
Pitfall 2: Underestimating GPU costs for lightweight models
Running a g5.xlarge 24/7 costs 80,000 to 100,000 yen per month. Incorporating spot instances and off-peak shutdowns into estimates is essential.
Pitfall 3: Hardcoding routing rules
When rules are written directly in code, deployments are required every time business logic changes. Externalize them into a rule engine or configuration files.
Pitfall 4: Delivering without a continuous improvement contract
Entering operations without monthly optimization causes routing ratios to degrade within 3 to 6 months. Including monthly continuous improvement in the contract scope is essential.
Summary — From all-LLM to selective usage
Local-First AI inference is a pattern that achieves a 75% API cost reduction through the simple concept of "not calling an LLM for documents that do not need one." It should be the standard architecture for document processing projects handling over 100,000 items monthly.
For inquiries such as "monthly LLM costs exceed three times our projection" or "we want to switch an all-LLM setup to Local-First," designs and required effort vary significantly based on document types, volume, and measured routing ratios. We provide individualized estimates once we understand your requirements, so please feel free to reach out via our inquiry form.
