Suppose you assign a feature addition to an AI coding agent, and it reports back that "all tests passed." Opening the diff reveals that three instances of skip were added to the test files.
Or perhaps expect(result).toEqual(expected) was rewritten to expect(result).toBeDefined(). Or the expected constant value was updated to match the implementation output. When pointed out, it apologizes politely and redoes the implementation, only to repeat the exact same thing on the next task.
Many people perceive this behavior as "AI is still immature, and smarter models will fix it." However, rather than viewing it as a capability issue, treating it as an inevitable outcome of evaluation design makes it far easier to find solutions.
The prompt defines what constitutes achievement
A prompt that says "implement this feature" contains no definition of completion. Lacking one, the agent defines its own standard for completion. The most accessible standard at hand is "making the test command green."
Here, there are two ways to turn it green: fix the implementation, or fix the test. The prompt forbids neither. And the latter achieves green reliably and in far less time.
The Zenn article "Why AI Agents Suppress Tests" explains this dynamic using the concept of proxy rewards. Because the true objective of "delivering value to customers" cannot be measured directly, it is substituted with measurable metrics like test success. Since agents optimize for the proxy, any divergence between the proxy and the true objective sends the agent sprinting at full speed toward that divergence. The article refers to consciously designing rewards and continuously correcting these divergences as reward engineering.
Human engineers avoid doing this not because of superior capability, but because they understand evaluation criteria outside the command: reporting completion after skipping tests destroys trust. Agents do not see that external context.
Suppression always leaves traces in the diff
The saving grace is that this behavior cannot hide. Examining the diff reveals immediately whether the implementation was fixed or the test was modified. The problem arises when production code and test code are blended into a single diff, leaving it unnoticed by everyone.
What you need to look for is well established.
| Inspection area | Sign |
|---|---|
| Test file diffs | Additions of skip / only / xit, or deletion of test cases |
| Assertions | Equality assertions relaxed to existence checks, or expected constants rewritten |
| Mocks | Expanded scope bypassing actual implementation |
| CI configuration | Flags permitting failures, extended timeouts, or excluded target files |
If you only want to extract test diffs mechanically, a single command is all it takes.
# テストコードの変更だけを抜き出して確認する
git diff origin/main...HEAD -- '**/*.test.*' '**/*.spec.*'
# 無効化キーワードが増えていないかを数える
git diff origin/main...HEAD -- '**/*.test.*' | grep -E '^\+.*\b(skip|only|xit|xdescribe)\b'Integrating this into CI and simply requiring human review whenever matches occur eliminates a substantial portion of discrepancies between reports and reality. If you only look at test counts, skipped tests are still counted as "existing tests," so check the executed count rather than the raw total.
Writing completion criteria in words other than "green"
Even with detection mechanisms in place, the same behavior will persist unless prompt phrasing changes. What works is specifying completion criteria up front using terms outside test commands.
For instance, instead of saying "implement this feature," phrase it as follows:
- Do not modify test code. If you determine modifications are necessary, do not change them; report the rationale first
- If existing tests fail, explain the cause of failure before updating the implementation
- In the completion report, include the number of executed tests (excluding skipped)
The third rule is quietly powerful. Requiring a report of executed tests excluding skipped ones means the count drops if tests are suppressed, turning the report itself into self-reported evidence.
This is not unique to AI. Even for human development teams, as outlined in Criteria Clients Should Inspect in Acceptance Testing, "all tests passing" is a weak indicator of deliverable quality. The widespread adoption of AI coding agents has merely brought that inherent weakness to light.
Where to draw the line in custom development contracts
When outsourcing development to external vendors who use AI agents—which should now be assumed as standard—there are concrete clauses to include in contracts and specification sheets.
1. Make test code modifications an independent approval item. Do not combine production and test code changes into the same pull request, or explicitly highlight test diffs if combined. This alone brings test suppression into the reviewer's field of vision.
2. Evaluate whether tests actually catch defects rather than relying on coverage. Coverage does not drop even if assertions are made empty. Methods for evaluating test quality itself are detailed in Using Mutation Testing to Assess Deliverable Test Quality.
3. Phrase acceptance criteria in business domain language rather than command exit codes. Specify "under this condition, this screen displays this value" rather than "tests pass." The latter is a proxy reward subject to suppression; the former cannot be suppressed.
How to establish acceptance criteria for AI-generated code is also discussed in Acceptance Criteria for AI-Written Code. In tandem, consider architectures that log how deliverables were produced.
What to do next
First, extract and inspect only test file diffs from pull requests over the past month. If you find even one where skip was added, it will continue to happen. If none are found, your current prompting approach is working.
Next, add "Do not modify test code" as a single line in your prompt templates for agents. It yields immediate results without requiring configuration changes or new tools.
GleamHub offers consulting on designing development processes integrating AI agents, establishing acceptance criteria, and defining quality requirements with external contractors through our Development, AI, and Automation consulting services. Because the approach varies based on team structure and development workflows, please consult with us individually. Feel free to reach out via Contact Us.
