"When we launched a conversational AI to answer questions about internal inventory and pricing, it reported inventory counts that did not actually exist, adopting a tone as if it had verified them." Almost without exception, this is the first wall companies hit when embedding AI into business processes. What makes it particularly pernicious is that it sounds entirely confident despite being wrong. Where a human would add "probably," the AI responds definitively, leading the recipient to trust it blindly.
These "plausible lies (hallucinations)" and occasional "rogue executions" do not happen because the AI is underpowered. The root cause is an architecture that entrusts both judgment and execution entirely to the large language model (LLM). In July 2026, food delivery platform DoorDash published the architecture of its conversational AI assistant, and its core principle boils down to: "do not rely on LLMs alone." This article breaks down why delegating everything is dangerous, and how to separate what to entrust to AI from what to keep external when creating or commissioning business AI.
Why "Leaving Everything to the LLM" Breeds Lies and Erratic Behavior
An LLM is an engine designed to generate plausible sequences of words given a prompt context. By default, it lacks human-like instincts such as looking things up when uncertain or verifying figures in a ledger. Consequently, when tasks involving information where accuracy is vital—such as inventory quantities or pricing—are left to generation, the model effortlessly invents "probable numbers." That is the true nature of hallucinations.
The same vulnerability applies to execution. If the determination of whether an operation is permitted is left to the LLM, it will trigger actions without validation, risking irreversible outcomes. This danger escalates as AI agents are granted broader permissions (an aspect we also examined in Guardrail Design: Lessons from an AI Agent Deleting Production Data). In short: the more precision and execution are delegated to an LLM, the more fragile the system becomes.
DoorDash's Answer — Offloading Parts Requiring Accuracy Outside the AI
DoorDash's architecture demonstrates a division of labor where the LLM is restricted to handling language, while tasks demanding accuracy are offloaded to deterministic systems outside the AI. In practice, it employs the following combination:
| Task | Assigned component |
|---|---|
| Operational logic such as inventory checks, price calculations, and cart actions | Deterministic logic (mechanisms that return identical results every time) |
| Targeted searches, such as finding products by exact brand name | Keyword search (transparent and fast) |
| Ambiguous queries, such as searching for similar items | Meaning-based search (semantic search) |
| Conversational phrasing and intent extraction | LLM |
Two design principles stand out. First, business logic is maintained as external tools (components) rather than inside the LLM prompt. Cart modifications and inventory lookups are encapsulated as deterministic modules that the LLM merely invokes. Second, hybrid search combines keyword and semantic approaches. Exact requests use precise keyword search, while ambiguous intents use semantic retrieval (our RAG Optimization Pattern Catalog offers helpful guidance on fine-tuning search). Crucially, confirmation checkpoints are embedded before significant actions to prevent the AI from executing tasks autonomously.
Checkpoints for Project Owners — Does the Architecture Specify "What Not to Delegate"?
When outsourcing business AI or developing it internally, what separates good deliverables from bad is whether the design explicitly defines what to delegate to AI and what not to. If this remains ambiguous and the system simply asks the LLM everything, it may work in demos but will inevitably lie and misbehave in production. When commissioning a project, confirm the following:
- Are figures requiring precision—such as inventory, pricing, and balances—retrieved from deterministic systems rather than generated?
- Does search distinguish between exact targets and ambiguous queries rather than routing everything through semantic search?
- Are there confirmation and approval checkpoints prior to operations that modify data or involve financial transactions?
- Is the system architected so the AI can say "I don't know" rather than forcing a response?
The most dangerous mindset is assuming that assigning everything to AI makes the system smarter. In reality, the opposite is true: the more you isolate precision-critical components outside the AI, the more dependable the system becomes. Evaluate proposals based on whether this boundary exists rather than how impressive the demo looks. For mechanisms that ensure final deliverable standards, reviewing QA Governance for AI Deliverables provides added peace of mind.
If you have deployed an AI chatbot but hesitate to release it to production for fear of errors, or if you want to automate internal inquiries and workflows while safeguarding against rogue operations, please feel free to reach out to GleamHub for custom development, AI, and automation consulting. From delineating boundaries between AI and deterministic logic to engineering resilient production architectures, we can build the system with you.
