In May 2026, the GitHub Blog published Validating agentic behavior when “correct” isn’t deterministic, presenting the Trust Layer and dominatory analysis as methodologies for "validating AI agent behavior in domains where there is no deterministic correct answer." In an agent era where "the same input can yield multiple correct answers," this represents an official acknowledgment that traditional unit testing cannot guarantee quality.
At our company, custom projects incorporating Claude Code, GitHub Copilot, Cursor, and Codex have become routine, making substantiated rationale to explain to clients that "code written by AI is correct" indispensable. This article breaks down the evaluation infrastructure, contract clauses, and operational dashboard design needed to integrate the Trust Layer into custom development.
Why is there "no single correct answer"?
Agent output fluctuates across the following three axes:
| Axis of fluctuation | Examples | Impact on custom development |
|---|---|---|
| Non-deterministic generation | Different code is produced even with the same prompt | Test reproducibility is difficult |
| Multiple correct answers | Multiple implementation approaches exist for the same specification | Review standards become inconsistent |
| Environmental dependencies | Behavior changes depending on OS and library versions | Verification costs rise |
The problem of multiple correct answers is particularly severe in custom development, posing the risk of churning out code that "passes tests but diverges from client intent." This challenge is two sides of the same coin as "how to measure the validity of tests written by AI," which we explored in Vitest 4.1's AI Agent Reporter.
The four tiers of the Trust Layer
The Trust Layer proposed by GitHub is an approach that, rather than judging output directly, observes the process and rationale leading to output across multiple layers. At our company, we configure it across the following four tiers for custom development:
[Layer 1: Behavior Trace（行動ログ）]
├ ツール呼び出しの順序・引数・戻り値
├ ファイル変更の差分（before / after）
└ 失敗 → 再試行の連鎖
[Layer 2: Property Tests（性質ベーステスト）]
├ 不変条件: 「データを失わない」「権限を逸脱しない」
├ 等価性: 「リファクタ前後で同じ入出力」
└ ベンチマーク: 「速度 / メモリの逸脱なし」
[Layer 3: Dominatory Analysis（支配関係分析）]
├ 候補解 A / B / C を生成
├ 「全評価軸で A が B 以上」を判定
└ 支配解のみを採用
[Layer 4: Human-in-the-Loop Sampling]
├ 高リスクな差分は人間レビュー必須
├ ランダム抜き取りで 5〜10% を二重チェック
└ 顧客 PM の最終承認ゲートIn particular, Layer 3: dominatory analysis is based on the concept of "filtering out demonstrably inferior solutions rather than selecting a single best one," making it a technique highly compatible with custom development code review culture.
Template for "AI quality clauses" in custom development contracts
As long as code written by AI is included in client deliverables, explicitly stating quality standards in contracts is the minimum prerequisite for preventing incidents. Below are key points of the clauses we standardize at our company:
| Clause | Details | What the client should verify |
|---|---|---|
| Liability for AI output | Our company assumes quality liability for final deliverables | Prohibition on providing data for model training |
| Verification metrics | Enumerate passing criteria across the four Trust Layer tiers | Whether thresholds for each tier are realistic |
| Scope of human review | Definition of high-risk diffs and designated approvers | Whether individuals with approval authority are designated |
| Reproducibility | Equivalent behavior can be reproduced with equivalent prompts | Log retention period |
| Model change notification | Prior agreement upon switching adopted models | Notification SLA |
| Provision of audit logs | Disclose logs upon client request | Duration and granularity |
Vaguely defining the "scope of human review" invites incidents where production breaks due to an automated merge on a Friday evening. Explicitly stating that "DB migrations, authentication modifications, billing processing, and code handling personal data require mandatory human approval" is the baseline operational standard.
Think of this as formalizing into contract terms the Approval Gates concept discussed in Safely Operating Claude Code Auto Mode in Custom Development.
Trust Dashboard — Operational screen shared with clients
Consolidating Trust Layer validation results into a dashboard that client PMs can review in 5 minutes daily is the key to sustainable operation. We display the following metrics at our company:
- Adoption rate: Percentage of agent proposals that were merged
- Rejection rate: Percentage rejected via dominatory analysis
- Human intervention rate: Percentage sent back via Human-in-the-Loop
- Verification time: Median and 95th percentile from proposal to merge
- Incident chains: Weekly number of rollbacks attributable to AI
- Quality score by model: Adoption and rejection rates per model
We visualize these via GitHub Audit Log + BigQuery + Looker Studio and operate them in a cycle of weekly reviews together with client PMs. Integrating this into the same dashboard as the cost visualization from GitHub Copilot Usage-Based Billing and Token Governance rapidly matures the practice of "managing cost and quality simultaneously."
Five common pitfalls
Finally, we share common pitfalls encountered when establishing a Trust Layer in custom development.
Pitfall 1: Assuming testing equals validation
Even if unit tests pass, they cannot detect "implementations that diverge from client intent." Run Layer 2 Property Tests and Layer 3 dominatory analysis concurrently from the very beginning.
Pitfall 2: Collecting logs without analyzing them
Cases frequently arise where behavioral logs are simply stored in Cloud Logging without ever being utilized. Incorporate a mandatory one-hour weekly block into operations to review logs.
Pitfall 3: Ambiguous criteria for human review
Under a vague standard like "important changes require human review," humans end up reviewing every PR, degrading productivity. Automate programmatic classification across three axes: file path, lines of diff, and touched modules.
Pitfall 4: Failing to update the Trust Layer when switching models
When switching models from GPT-5.5 to Claude, keeping thresholds and Property Tests unchanged causes validation to break down. Make switching models trigger Trust Layer recalibration as an operational rule.
Pitfall 5: Client PMs not looking at the dashboard
An operational workflow where our company alone monitors quality degrades into a contract where "our company unilaterally warrants quality." Include a contractual commitment for client PMs to spend 5 minutes reviewing weekly, turning the dashboard into a space of shared responsibility.
Summary — Quality assurance in custom development in an "era where tests alone are not enough"
With the spread of AI agents, "tests pass = correct" is no longer sufficient as quality assurance in custom development. GitHub's Trust Layer offers the most implementable answer today to the question of "how to trust outputs when there is no single deterministic correct answer," making it well worth adopting as a quality assurance framework in custom development.
How many tiers of the Trust Layer to construct depends heavily on the risk profile of the code being handled and your existing CI/review structure. If you are struggling with issues like "not knowing how to explain AI-written code to clients" or "post-merge rollbacks failing to decrease," we provide individual estimates after assessing your current organization. Feel free to contact us via our inquiry form.
