An organization commissioned an internal inquiry AI to answer questions about company regulations. During the demo, it answered every question asked correctly. Two months after deployment, reports came from General Affairs: "It occasionally says incorrect things, so we end up opening the regulations manual to check anyway."
When reported to the development vendor, an updated version arrived two weeks later. This is where things stalled. Even when told "it's fixed," the internal team had no way to confirm whether it was actually fixed. All they could do was test a few of the earlier questions, and if the AI happened to answer correctly, conclude that "it must be fixed."
This is not due to negligence by the development vendor; it is the direct consequence of not defining "what constitutes correctness" at the time of procurement.
What looked perfect in the demo falls apart in production
Features powered by generative AI behave differently from conventional systems. The same input does not guarantee the exact same output, and functionality cannot be judged by a binary "working / broken" test.
GitHub's published guide on production-grade evaluation outlines the reasons behind this discrepancy. It is not uncommon for a model that achieved strong results on clean benchmarks to miss critical edge cases in production. The factors highlighted there apply directly to internal business systems as well.
- Actual user inputs are ambiguous and feature inconsistent phrasing
- Determining what constitutes the correct answer can itself vary among human reviewers
- Necessary context may be missing or truncated midway
- Data used for evaluation does not reflect the distribution of real-world production queries
- Edge cases rarely seen in benchmarks become the primary causes of failure in production
Demos are conducted using "questions the creators already know the model can answer." Real-world queries fall outside that scope. Bridging this gap requires structured evaluation, which is not an afterthought to tack on at the end of development, but a system that must be designed during procurement.
Only the client knows the correct answers
This is the most critical practical point: the party capable of judging "what is the correct answer to this question" is the client who knows the operational workflows. The development agency does not understand the day-to-day realities of your company policies.
Therefore, the evaluation dataset must be prepared by the client. It is not a difficult task.
- Gather actual past inquiries — Collect 50 to 100 real inquiries from the General Affairs or IT team inbox and chat logs. Do not fabricate them. Real phrasing contains typos and omissions, and that reflects your actual production inputs.
- Have operational staff write the correct answers — Short answers are fine. A level of detail like "According to Article 12 of the Work Regulations, applications must be submitted by the previous day" is sufficient.
- Mix in questions the AI should not answer — Queries requiring discretionary judgment, such as "Will this expense be approved?", should not be answered by an AI. Returning "I don't know; please check with the person in charge" is the correct answer, and should be registered as such.
Without the third type of question, you cannot detect when the AI fabricates an answer. In business operations, the real hazard is not a system that stays silent when it cannot answer, but rather a system that returns plausible hallucinations.
Before putting "90% accuracy" into the contract
Numerical targets are straightforward and easy to write into contracts. However, making a single metric an acceptance condition means work will focus strictly on driving up that number. This follows the exact same pattern discussed in acceptance testing for AI-generated rapid code regarding score-based acceptance conditions.
Suppose you have an evaluation set of 100 items with a 90% passing threshold. The development team will make adjustments to pass the remaining 10 items. If those adjustments represent generalized improvements, all is well; however, if they are merely hard-coded fixes targeting only those 10 cases, the score alone cannot reveal the difference.
In practice, structuring acceptance criteria as follows is far more effective.
| Decision Item | Specification Example |
|---|---|
| Evaluation set | 100 questions provided by the client, 20 of which are questions that "should not be answered" |
| Passing criteria | Overall accuracy target, plus "zero incorrect answers among the 20 questions that should not be answered" |
| Evaluation execution | Re-run the same dataset at delivery and after every modification, submitting the full test results |
| Dataset updates | Add incorrect answers discovered during production operations to the evaluation set |
The second and third rows are what make this work. If you rely solely on an average score, dangerous hallucinations simply blend in as minor point deductions. By isolating "questions where an incorrect answer is unacceptable" with a mandatory zero-tolerance threshold, critical failures cannot hide.
Furthermore, the third row resolves the dilemma of "not being able to confirm whether it was fixed." Because the same dataset is run every time, you can immediately verify whether a fix broke something else elsewhere.
The real test begins after operations launch
The evaluation dataset is not finished upon delivery. Continually adding incorrect responses uncovered during real-world operations will bring it closer to actual usage patterns.
When front-line staff report that "the AI was wrong," make sure to record the prompt, the output provided, and the correct answer. Without records, feedback remains vague impressions like "it makes mistakes sometimes," and modification requests will be communicated just as vaguely. Even just three documented cases give the development agency what they need to reproduce the issue.
At this stage, you may also decide to re-evaluate what scope should be delegated to AI. For inquiries requiring human judgment or involving financial figures, designing the workflow to hand off to human staff rather than having generative AI answer is far more reliable. Our perspective on how much to shift toward deterministic rules is outlined in Don't build business AI with generative AI alone.
What to do next
First, gather just 30 past inquiries. Whether development has begun or not, this can be done today. As soon as you collect them, the distinction between questions suited for AI and those unsuited will become clearly visible.
Next, if you already have an active AI feature in production, set up a dedicated place to record "reported incorrect answers." A single spreadsheet is plenty. When making your next update request, whether you have this log will completely transform the discussion.
GleamHub offers development, AI, and automation consultations to assist with AI requirements definition, evaluation design, and integration into existing systems. Because system design depends on the nature of your business operations and required accuracy, please contact us for individualized guidance via our inquiry form.
