The demo worked. The agent answered questions from staff, referenced internal documents, and executives agreed: "This is great."
Six months later, talks of moving to production have made no progress. When you ask why things stalled, it usually starts with the preface: "It's not that the accuracy isn't good enough, but..." A state where no one opposes the initiative yet nothing moves forward is not a technical problem; it is a situation where undetermined requirements remain.
Here, we assume a scenario where the demo proved technically sound, but production operational conditions were never settled, leaving things stalled. Because insufficient accuracy, data quality, and budget can also be culprits, these four items alone do not explain every case.
Stalling before discussions on accuracy even begin
When companies in highly regulated industries adopt AI agents, they inevitably face a choice: build in-house or purchase an off-the-shelf product. In a series on gihyo.jp, the article points out how internal tools become fragmented even before reaching that decision.
Different departments adopt disparate tools, each with its own permission model and history. When introducing an agent into this environment, debates over "which data can be shown" occur as many times as there are departments. Even if a PoC succeeds in one department, the second department forces you to restart that same debate from scratch.
In other words, what the PoC verified was "whether the agent can answer," not "whether it can be operated across the organization." Conflating the two before placing an order means paying for a working demo, and that is where it ends.
4 things to decide before ordering
There are four items that, when clarified during requirements definition, enable tangible decisions about moving to production.
1. How to measure whether a task is "completed"
Do not rely on the agent's self-reporting as its success rate. Whether a task was executed to the end and whether it avoided doing unintended things during execution are two separate metrics.
If an agent assigned to generate quotes completes the document but inadvertently emails it to a client, the task itself was technically finished. What should be measured is not "completion rate," but "side-effect-free completion rate."
This definition must be set by the ordering party. Only those actually running the business operations can define what constitutes business success.
2. How much runs automatically, and where humans approve
Permissions granted to an agent differ significantly: read-only, drafting only, or full execution. Building without drawing this line will later require modifications to restrict permissions, a scope comparable to rebuilding from scratch.
The criterion for drawing the line is whether an action can be rolled back upon failure. Generating a draft can be redone. Sending an email, reserving inventory, or making an external payment cannot be reversed. Placing human approval immediately before irreversible operations serves as the starting point.
Furthermore, when providing an execution environment for an agent, you must simultaneously determine the boundaries of its network access. Execution environment isolation and delegated authority architecture are difficult to retrofit later.
3. Who prepares the evaluation data
This is the item most frequently overlooked.
To continually verify that an agent operates correctly, you need pairs of expected inquiry or task patterns matched with their anticipated outcomes. In practical examples highlighted by InfoQ, synthetic user personas simulate multi-turn conversations, integrating evaluations directly into CI/CD pipelines.
If developers invent this evaluation data from imagination, it will deviate from operational realities. The fastest route is for the ordering party to provide representative cases from past inquiry histories and operational logs. If an estimate lacks a line item for "evaluation dataset creation" and no request has been made to the client, that evaluation will inevitably be skipped somewhere.
4. Whether operations can revert if the agent is stopped
The decision to shut down an agent will inevitably arise. The reason may not be accuracy issues, but external service outages or changes in pricing models.
When you turn it off, do standard operating procedures still exist to run the task manually? If you discard existing runbooks when introducing an agent, you will no longer be able to stop it. A system you cannot turn off is one you have no choice but to keep running, even under worsening terms.
Line items to look for when reviewing quotes
The four items above translate directly into checklist items when reviewing quotes.
| Decision Item | Line items to look for in a quote |
|---|---|
| Definition of done | Inclusion of acceptance criteria and test cases |
| Permission boundaries | Approval workflow and execution permission design |
| Evaluation data | Creation of evaluation datasets, or explicit designation as client-side work |
| Rollback | Operational runbooks and preservation of existing procedures |
A quote lacking these lines that merely lists features alongside labor hours is a quote for a PoC, not for a production system. The ordering party needs to make clear which one is being requested.
Ordering in phases is a sensible approach. In that case, however, ensure that the deliverables of the first phase include the materials needed to evaluate the second phase. If only a demo remains, the next decision cannot be made. For writing requirements themselves, how to summarize system development requests serves as a useful reference.
Common pitfalls
Do not begin a PoC without defining success criteria. Validations started with "let's just give it a try" cannot be judged as successes or failures when they conclude. Without that verdict, you can neither decide to proceed nor decide to stop, and everything grinds to a halt.
Estimate operational costs separately from initial expenses. Usage-based fees, evaluation updates, and adaptations to model or tool changes will occur continuously once live. The pattern of operational costs exceeding projections is common to AI-integrated systems across the board.
Do not postpone cross-departmental discussions. Starting focused within a single department is the right approach, but knowing that permission debates will occur with the second department is predictable from day one. Define the overarching policy on permissions at the company-wide level while still in the first department.
What to do next
For the agent you are currently considering, write down in a single sentence: "If it can accomplish this, we will move to production." If you cannot write it, that is the reason things are currently stalled.
Once written, add who will measure that condition and with what data. When all three are aligned, the ownership and scope of verification for production readiness become concrete.
At GleamHub, we provide consultations on development, AI, and automation to assist with scoping business requirements, designing boundaries between approval and autonomous execution, and developing systems that include evaluation frameworks. Because approaches vary based on business operations and existing workflows, scope is estimated individually. Please reach out via Contact Us.
