"Our crawler traffic in server logs tripled last month, with most requests identifying as ClaudeBot and Googlebot. We allow them under our policy to support AI indexing, but is this volume normal?" Site operations teams taking over existing web properties often bring us questions like this.
In August 2026, the answer to this question shifted. Large-scale activity has been observed where vulnerability scans disguise themselves under AI crawler User-Agents. Attackers frequently impersonate ClaudeBot or Googlebot, targeting lingering credentials left behind in rushed deployments of AI coding tools or internal LLM environments.
Anyone can claim any User-Agent
Let us establish a fundamental premise: A User-Agent is arbitrary text defined by the requester and offers zero proof of identity. Anyone can claim to be ClaudeBot/1.0 with a single curl flag. This is by protocol design, not a software vulnerability.
The vulnerability lies in operations teams treating this header as proof of identity. Many organizations established crawler policies over the past two years, typically adopting one of two approaches:
- Explicitly allowing (or blocking) AI crawlers in robots.txt
- Writing bypass exceptions into WAF or CDN bot protection rules conditioned on User-Agent string matches
To attackers, the latter represents a bypass route with relaxed inspection. Furthermore, operational monitoring dashboards aggregate that traffic under benign crawler metrics. Because malicious requests do not surface as anomalies, detection takes significantly longer.
It is equally important to remember that robots.txt is completely advisory. While Anthropic explicitly adheres to robots.txt directives, that policy applies exclusively to legitimate crawlers; malicious actors spoofing crawler names have no incentive to comply.
Methods for verifying genuine crawlers
Instead of relying on claimed identities, inspect the network source directly. Two primary verification methods exist, depending on provider capabilities.
First is cross-referencing published IP ranges. Google publishes machine-readable JSON feeds of crawler IP ranges, allowing systems to programmatically verify whether traffic claiming to be Googlebot originates within authorized blocks. Where available, this provides the most reliable verification.
Second is reverse DNS validation. This involves running a reverse DNS query on the connecting IP to resolve its hostname, followed by a forward DNS query on that hostname to verify it maps back to the original IP. This bidirectional validation is known as forward-confirmed reverse DNS (FCrDNS). Reverse DNS alone is insufficient; verification is incomplete unless forward resolution successfully confirms the original IP. Some implementations mistakenly perform only the initial reverse lookup.
# 逆引き（IP → ホスト名）
$ dig -x 203.0.113.10 +short
crawler-203-0-113-10.example-vendor.com.
# 正引きで戻るか（ホスト名 → IP）
$ dig +short crawler-203-0-113-10.example-vendor.com
203.0.113.10Because provider policies on publishing IP blocks can evolve, consult official documentation at the time of implementation. For providers that do not publish IP blocks, bidirectional FCrDNS serves as your practical baseline.
Looking further ahead, Cloudflare has proposed Web Bot Auth. Built on HTTP Message Signatures, this framework allows agents to cryptographically verify their identity on every HTTP request. Spurred by concerns that rapid proliferation of AI agents makes maintaining IP allowlists unsustainable, this standard represents the likely medium-to-long-term direction for web verification.
Actionable checks for operations teams today
Before investing in complex new architectures, teams can immediately close vulnerabilities in existing rules.
1. Check whether bot allow rules on your WAF and CDN rely on User-Agent matching. If there is a condition stating "allow if it contains ClaudeBot," that serves as an open bypass path. If your CDN provides verified bot detection capabilities, switch from string matching to that feature.
2. Verify whether bot exclusion in your analytics is broken. If spoofed requests are being filtered out as crawlers, not only are you unable to observe the damage, but it also proves that your exclusion criteria depend on the same string. Conversely, if they are counted as human visits, page views and session counts will diverge from reality.
3. Examine bandwidth and origin reach. If traffic allowed under crawler rules is reaching your origin server, it may be following paths that bypass CDN caching. Beyond being a cost concern, this also means exploratory requests are directly hitting your application.
4. Inspect the targets being probed. Spoofed scans typically target .env, .git/config, administrative paths, and exposed configuration files. If your logs show an increase in 404s for these paths, that is not normal crawler behavior. Do not treat high volumes of 404 errors as noise you can safely ignore. The only record of what attackers came looking for remains inside those 404 details.
Note that this concerns verifying the identity of incoming traffic, which is a separate matter from deciding whether to treat legitimate AI agents as customers. Perspectives on distinguishing intent from visitor behavior are discussed in AI Agents: Customers or Bots?, while access policies for AI crawlers themselves are covered in AI Crawlers and Content Sovereignty.
What to do next
First, check whether your site's WAF or CDN contains allow rules based on User-Agent strings. If such rules exist, verify how they behave against this tactic. If you are permitting traffic through string matches, replace them with verified bot checks, or at least restrict the scope of exceptions to static files.
Next, aggregate 404 responses from accesses claiming to be crawlers in your logs from the past month. If nonexistent configuration files and administrative paths line up, your site is undergoing probing rather than indexing.
If you would like consultation on analyzing access logs or designing bot controls across CDNs and WAFs, we assist with this through GleamHub's Development, AI, and Automation consulting services. Because viable approaches vary depending on your architecture and current services, please consult with us individually. Reach out via Contact Us.
Sources
- Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot — Hacker News
- Anthropic clarifies what its three web crawlers do - and how to block them — PPC Land
- Verifying Googlebot and other Google crawlers — Google Search Central
- AI Bot Verification and Edge Enforcement: 2026 Playbook — Digital Applied
- The AI User-Agent Landscape in 2026: A Complete Reference — No Hacks
