"After automating invoice classification with AI, in one specific month items were mistakenly assigned to another department's expense category, which was discovered during the financial close. Is this the developer's responsibility, or the user's?"—After handing over systems incorporating AI features, such inquiries are not uncommon. The client believes, "Since you contracted the automation, you should be accountable for errors as well," while the vendor believes, "AI decisions are probabilistic, and we never intended to guarantee 100% accuracy." Facing the same contract, their fundamental premises do not mesh.
At the root of this disconnect lies the inherent nature of AI technology. Traditional systems were "correct if they behaved according to specifications," but because AI acquires behavior inductively from training data, it is fundamentally difficult to guarantee accuracy for unseen inputs in advance. That is precisely why the Ministry of Economy, Trade and Industry's "Contract Guidelines on AI and Data Utilization" proposes a phased approach—Assessment, PoC, Development, and Additional Training—recognizing that AI development cannot be treated like traditional work-for-hire. Rather than focusing on "how to proceed with AI development," this article addresses the preceding question: when damages arise from AI malfunction or erroneous classification, how do the client and vendor divide responsibility, and how should boundaries be established across contract, architecture, and operations? We organize this from the perspective of handling custom development.
The Premise of "Contracted = Result Guaranteed" Breaks Down With AI
First, we must examine contract types. Contracts in system development broadly divide into completed-contract models (contract for work) and quasi-delegation models. A completed contract assumes liability for "completion of deliverables"; if the finished product fails to satisfy agreed-upon quality, non-conformity liability (formerly defect liability) applies. Quasi-delegation assumes responsibility for "performing duties with the care of a prudent manager," without guaranteeing the result itself.
For standard systems, "a completed contract to build to specification" suffices. The problem is whether AI accuracy can be subject to a completed contract. When accuracy depends on training data quantity and quality, and production inputs cannot be exhaustively predicted in advance, it is unrealistic for a vendor to "guarantee a 99% accuracy rate as a deliverable obligation." In practice, teams frequently use different contract models within the same project—applying quasi-delegation to the core AI components (best-effort pursuit of specified accuracy) and completed contracts to surrounding standard software.
| Contract Type | Liability Borne | Suitability for AI Features |
|---|---|---|
| Completed Contract (Work for Hire) | Deliverable completion, non-conformity liability | Suited for standard software components; unsuited for guaranteeing AI accuracy |
| Quasi-delegation | Duty of care of a prudent manager (process). Results are not guaranteed. | Suited for AI model generation and tuning |
The greatest danger is when the agreement specifies a completed contract, yet contains no consensus regarding AI accuracy. The client reads it as "a completed contract includes results," while the vendor reads it as "being AI, it represents a best-effort duty." This gap in interpretation surfaces only after delivery when trouble arises. Therefore, in projects involving AI, it is critical to state explicitly in the contract where contract types switch, and what is guaranteed versus excluded for AI components. We touched on how reasons behind architectural decisions become unclear during handoffs in our article on organizing domain knowledge placement, and responsibility boundaries are a prime example of historical rationale being lost later.
Drawing Lines in Contracts — What to Guarantee and What to Exclude
Drawing lines at the contract level ultimately involves explicitly spelling out guaranteed scopes and excluded scopes. The "Checklist for Contracts Concerning AI Use and Development" published by METR in 2025 similarly urges parties to confirm points like risk allocation, accuracy handling, and data rights before signing. In custom development, clearly stating the following three points proves particularly effective.
First, treat accuracy as a "target metric" rather than a "guaranteed value". In SLA practice, inaccurate AI output is viewed as performance variance; promising financial compensation for missing accuracy targets is considered untenable. Practically, remedies are structured around best-effort obligations for retraining and model tuning, or service credits. Rather than writing "guarantees an accuracy rate of X% or higher," specify "targets X%, and in the event of shortfall, will strive for improvement through retraining." This single wording difference fundamentally alters whether liability for damages exists later.
Second, specify exclusion clauses concretely. Because AI output heavily depends on input data quality, explicitly stipulate that degradation in accuracy caused by skewed or inaccurate client-provided data falls outside vendor liability. This not only protects the vendor, but also prevents unproductive finger-pointing over whose input caused accuracy shortfalls.
Third, set a liability cap. In model terms from the Japan Information Technology Services Industry Association (JISA), liability caps are commonly set around one month's service fee, excluding special damages and lost profits. When an AI misclassification cascades into major secondary damages (accounting restatements, third-party compensations), setting a liability cap prevents the vendor from bearing unlimited exposure and is essential for balancing risk against compensation.
However, note that simply including caps or disclaimers does not provide absolute peace of mind. Limitation of liability clauses may be judged inapplicable in cases of gross negligence by the vendor. One should understand that disclaimers are not a license to do anything, but rather an agreement on the upper limit of residual risk after building conscientiously.
Drawing Lines in Architecture — Where to Place the Boundary That Reverts to Humans
Delineating liability in contracts alone cannot prevent real-world operational damages. Where custom development vendors provide true value is architecting systems so that misclassifications do not become fatal in the first place. The linchpin here is human-in-the-loop (HITL).
HITL is an architectural principle that embeds human review and intervention into AI decision processes, serving to ensure AI accountability and safety. The key is not merely inserting a human, but designing boundaries based on operational risk to dictate which classifications proceed automatically and which route back to humans. If humans verify everything, automation is meaningless; if everything passes automatically, misclassifications translate directly into damages. The middle ground must be tailored to each project.
In practice, tiered architecture using AI confidence scores works effectively.
| Confidence Score | Processing Flow | Locus of Responsibility |
|---|---|---|
| High | Finalized automatically; sampled for post-hoc audits | Vendor designing the mechanism + Client running the audit framework |
| Medium | Human reviews, then approves or rejects | Client operator making the final determination |
| Low | Do not process automatically; always route back to human | Client (AI declines to make a determination) |
The strength of this design is that beyond being a technical safety mechanism, it makes the locus of responsibility visible. If low-confidence cases route back to humans, subsequent errors become decisions made under human review rather than AI erring autonomously. Conversely, if a misclassification occurs within an area deemed high-confidence, it becomes an issue for the system designer, prompting retraining or threshold review. Establishing explicit boundaries avoids retroactively disputing fault when incidents occur.
Furthermore, preparing fallbacks for malfunctions directly impacts architectural boundaries of responsibility. When an AI classification API becomes unresponsive or returns anomalous output, does the system silently adopt erroneous values, or does it fail safely into a human review queue? Without deciding this, system failures directly become operational damages. AI outputs should be treated defensively as uncertain external responses that can fail at any time, with explicit fail-safe boundaries. Much like our article covering pitfalls of premature generalization, where rushing to abstract creates fragility, fallbacks are safer when prioritizing direct paths that reliably revert to humans over complex mechanisms.
Drawing Lines in Operations — Logs and Audits Become "Evidence"
The third layer is operations. Regardless of how well lines are drawn in contracts and architecture, if a misclassification occurs and records do not show when, for what input, what the AI returned, and who handled it, determining responsibility devolves into mutual accusations.
What proves effective here is an operational design capturing classification logs and human intervention history. By recording AI inputs, outputs, confidence scores, and whether humans approved or rejected them, you can demonstrate factually during disputes whether items were routed back to humans as designed and whether actions were based on human review. In discussions regarding civil liability in AI utilization, whether causal links and responsibility can be proven remains central; logs ultimately underpin that proof. For clients, this provides the added benefit of catching AI accuracy degradation early through sampling audits.
In an automated classification system for internal documents developed by our firm (industry and company name withheld), these three layers of boundaries proved decisive. While the initial request was to automate classification entirely with AI, misclassifications carried direct risks of confidential data leakage. We explained that we could not accept a completed contract guaranteeing full automation accuracy. Instead, we agreed on quasi-delegation for the AI portion with target accuracy levels, and architected documents below a specific confidence threshold to always queue for operator review. Only high-confidence items were auto-finalized, supported by monthly sampling audits. Furthermore, input documents, classification results, confidence scores, and operator decisions were logged exhaustively.
After operations began, a misclassification was indeed discovered. Tracking the logs revealed that while confidence was rated high, the document had an unprecedented format. The issue was resolved by adding that format to training data and adjusting thresholds. Crucially, this incident never devolved into disputes over fault. Because boundaries between automation and human intervention were agreed upon in the contract and logs demonstrated operations adhered to design, teams moved straight into root-cause analysis and improvement. As discussed in our article on legacy migration with AI assistance, such agreements on responsibility boundaries are especially vital when grounding major overhauls on AI.
Where to Begin Alignment as a Custom Development Partner
The phrase "we can automate it with AI" is both a technical statement and a management decision regarding who bears what risks. What vendors need is not to overstate capabilities, but to concretely illustrate boundaries between what can and cannot be guaranteed across contract, architecture, and operational layers. For clients, having those boundaries clearly defined acts as a safety valve, preventing trouble and avoiding quagmires when issues occur.
As a first step, for AI automations currently under consideration or in operation, we recommend writing down on a single sheet of paper: "If a single misclassification occurs, who notices, who bears responsibility, and how is it corrected?" If that sheet remains blank, it is a sign that lines have not yet been drawn across contract, architecture, or operations.
If you are considering AI automation but have not sorted out misclassification risks and responsibility, if a vendor proposed automation without clarifying guarantees, or if you wish to review responsibility boundaries for existing AI features across contracts and design, please contact GleamHub. After hearing your requirements, we will help you establish practical boundaries through to implementation, defining what to automate, where to involve humans, what to guarantee in contracts, and what to exclude.
Sources
- Checklist for Contracts Concerning AI Use and Development (February 2025) － METI
- Contract Guidelines on AI and Data Utilization (June 2018) － METI
- Choosing Contract Models for Custom AI Development: Completed Contracts vs. Quasi-Delegation － JPO IP BASE
- 2nd Study Group on Civil Liability in AI Utilization: Secretariat Briefing Material (October 2025) － METI
- Is Your SLA Truly Functional? A Lawyer's Guide to SLA Legal Risks and Design Essentials － YS Law Office
- Human-in-the-Loop (HITL) － Resaro
