HN Debrief

Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot

  • Security
  • Infrastructure
  • AI
  • Developer Tools

Known Agents published data showing a sharp recent increase in automated scans that claim to be AI crawlers like ClaudeBot while requesting suspicious paths associated with exploitable apps and configs. The core claim is not that mass scanning is new. Internet-facing services have been hammered by commodity probes for decades. The novelty is the disguise. Attackers are borrowing the identities of AI bots and other legitimate crawlers because those names are less likely to get blocked outright and because some of the targeted paths line up with newer AI tooling.

Treat crawler identity as something to verify, not trust from a header. If you run public web services, tighten setup before exposure, watch certificate and DNS-triggered discovery, and expect attackers to blend into bot traffic you may currently allow.

Discussion mood

Mostly unsurprised and battle-hardened. People agreed the surge and AI-bot spoofing are real, but saw them as the latest wrapper on an old fact of life: anything exposed to the public internet gets scanned immediately, and defense is about verification and filtering rather than outrage.

Key insights

  1. 01

    Certificate Transparency exposes fresh targets

    New domains and newly issued TLS certificates get surfaced almost instantly through Certificate Transparency logs, which gives scanners a ready-made feed of fresh hosts to probe. That turns setup timing into a security issue. If a WordPress install or admin surface is reachable before lockdown, bots can often touch it before the owner does.

    Secure the service before issuing the public certificate or switching DNS. Put auth, firewall rules, and default-deny behavior in place first, then make the hostname visible.

      Attribution:
    • mmh0000 #1
    • esseph #1
    • jasonjayr #1
    • dawnerd #1
  2. 02

    ASN scoring beats simple bot blocking

    Classifying traffic by ASN, network type, and abuse signals gives you a more usable control plane than chasing fake crawler names. One operator described combining ASN categories, endpoint patterns like repeated .php probing, and account-creation signals to decide when to challenge with Turnstile and when to hard block. That reportedly cut a meaningful share of traffic without punishing authenticated users.

    If bot load is hurting the business, build or buy network reputation data and use it in risk-based rules. Keep challenges conditional so your logged-in users are not collateral damage.

      Attribution:
    • cullenking #1
  3. 03

    Overblocking can damage legitimate crawling

    Aggressive anti-bot tools can easily block the real Google, Bing, or OpenAI crawlers along with the fakes. That wastes crawl budget and can delay reindexing of updated pages. Verifying real crawlers against published IP lists is a cleaner approach than broad bot-fight settings that guess wrong.

    Audit your anti-bot setup for search and AI crawler false positives. If discovery matters to you, whitelist verified crawler IP ranges instead of relying on generic bot-defense modes.

      Attribution:
    • ChillyCapy #1
    • dewey #1
  4. 04

    The recent wave appears centrally coordinated

    Reports of synchronized traffic swings across thousands of IPs, with heavy volume from Google Cloud address space, point to a campaign under centralized control rather than random background noise. The use of many cloud hosts at once matters more than any single source IP because it makes manual block-and-report workflows ineffective.

    Prepare for bursty campaigns that rotate across cloud infrastructure. Rate limits, ASN-level controls, and automated reputation updates will age better than hand-maintained IP blacklists.

      Attribution:
    • oasisbob #1
    • gavinhking #1
  5. 05

    Web Bot Auth is becoming practical

    Crawler verification is starting to move beyond reverse DNS checks and IP ownership lookups. Google’s Web Bot Auth and similar mechanisms from crawler operators give sites a way to confirm that a request really came from the bot it claims to be. That directly addresses the exact weakness this scan campaign exploits.

    Track bot-auth support from the crawlers you care about and wire verification into your edge stack. The sooner you separate verified bots from everything else, the less valuable user-agent spoofing becomes.

      Attribution:
    • gavinhking #1 #2
    • wilg #1

Against the grain

  1. 01

    This is mostly old junk with new branding

    Mass vuln scanning long predates generative AI, from Code Red onward, and several people said most of the requested paths and behaviors look like the same background abuse operators have always seen. The AI angle may be real, but it does not change the base rate enough to justify treating this as a fundamentally new threat model.

    Do not let an AI-flavored label distort your priorities. Keep investing in the same basics that stop ordinary scanners, because they still do most of the work here.

      Attribution:
    • yabones #1
    • hluska #1
    • walrus01 #1
    • KomoD #1
  2. 02

    Sites may hesitate to block AI crawlers

    Pretending to be an AI bot is not necessarily irrational if many websites now want those crawlers to pass. Businesses increasingly see chatbot referrals as an emerging traffic source, and one commenter said those visits convert better than organic search even if the volumes are still smaller. That makes blanket blocks politically harder inside organizations.

    Expect internal pressure to allow at least some AI crawlers. Build verification and policy controls now so marketing goals do not force you into trusting spoofable identities.

      Attribution:
    • Tharre #1
    • esskay #1
    • xgulfie #1
    • cullenking #1
  3. 03

    Country-based attribution is shaky

    Treating attack traffic as definitively Chinese or Russian based on source IP can be misleading because routing and packet injection make simple geographic blame fragile. Even when geo-IP is operationally useful for blocking, it is weak evidence about who is actually behind a campaign.

    Use geo-blocking as a coarse filter if it helps, but do not confuse it with attribution. Base investigations and executive reporting on stronger signals than source-country labels.

      Attribution:
    • bflesch #1
    • pixl97 #1

In plain english

ASN ↩
Autonomous System Number, an identifier for a network or group of IP addresses managed as a unit on the internet.
Certificate Transparency ↩
A public logging system for TLS certificates that lets anyone see when a domain gets a new certificate.
Cloudflare Turnstile ↩
A bot-challenge system from Cloudflare used to distinguish likely humans from automated traffic.
crawl budget ↩
The limited amount of pages and request frequency that a search engine or crawler will spend on a site.
DNS ↩
Domain Name System, the internet service that translates domain names like facebook.com into network addresses.
reverse DNS ↩
A lookup that maps an IP address back to a hostname, often used as one weak signal for crawler verification.
TLS ↩
Transport Layer Security, the standard protocol used to encrypt web traffic such as HTTPS.
User-Agent ↩
An HTTP header where a browser or bot identifies itself by name and version, which can be easily faked.
VPN ↩
Virtual Private Network, a service that routes traffic through another network for privacy, policy bypassing, or location masking.
Web Bot Auth ↩
A mechanism that lets a crawler cryptographically or otherwise verifiably prove it is the bot it claims to be.
WordPress ↩
A widely used website and blogging platform that is often targeted because many sites run outdated plugins or configs.

Reference links

Crawler verification and bot auth

Certificate Transparency and certificate monitoring

Bot blocking tools and honeypots

IP and network reputation data

Background and related context

  • Code Red worm
    Historical example used to argue that mass automated scanning is an old feature of the internet
  • Darknet Diaries Episode 172
    Cited in the discussion about why IP blacklists alone fail when infected consumer devices share an IP with legitimate users