HN Debrief

Cloudflare's new AI traffic options for customers

  • AI
  • Infrastructure
  • Privacy
  • Regulation
  • Developer Tools

Cloudflare’s post introduced a new way for customers to classify and control AI traffic by behavior rather than a simple “AI bot” label. The key practical change is that new domains will block training and agent categories by default on ad pages, while multipurpose crawlers that mix search and training will inherit the most restrictive rule. That means customers who block training can end up blocking Googlebot, Applebot, and BingBot too. Cloudflare frames this as leverage for transparency and future monetization, including its still-not-generally-available pay-per-crawl idea.

If you run sites behind Cloudflare, test these settings before turning them on broadly because blocking “AI training” can also hit search indexing and referral traffic. Strategically, this is the early shape of a new web tollbooth layer, so founders should expect crawl access, attribution, and bot identity to become product and business issues, not just ops settings.

Discussion mood

Mostly wary and frustrated. People like having more control over AI crawlers, but the dominant reaction was distrust of both Google’s mixed search-and-training behavior and Cloudflare’s growing role as a private gatekeeper that can break traffic, block users, and eventually tax access to the web.

Key insights

  1. 01

    Many Googlebot incidents are spoofing problems

    A lot of the horror stories about Googlebot hammering sites may actually be fake bots borrowing the user agent string or Google Cloud IP space. That changes the operational response. Instead of treating every “Googlebot” spike as a search monopoly abuse issue, verify requests with reverse DNS, Google’s published guidance, or Cloudflare bot validation before you tune rules or panic about delisting.

    Separate “real crawler” traffic from impersonators in your logs before you change crawl policy. If you are behind Cloudflare, add rules that block Googlebot user agents unless Cloudflare recognizes them as verified bots.

      Attribution:
    • bobbiechen #1
    • simondotau #1
    • 20k #1
    • weird-eye-issue #1 #2
    • remus #1
  2. 02

    Custom heuristics can beat generic bot switches

    One operator described getting abusive traffic close to zero with simple Cloudflare rules based on URL length, path patterns, and a cookie that only gets set after visiting shallow pages first. The point is not that this exact recipe is universal. It is that site-specific behavior checks can outperform blanket “AI” or “bot” toggles because abusive crawlers often reveal themselves through traversal patterns rather than identity.

    Build defensive rules around how your real users move through your site. Start with cheap signals like path depth, asset loading, cookies, and API exceptions before you rely on one global anti-bot setting.

      Attribution:
    • nirui #1
    • inigyou #1
  3. 03

    Pay per crawl is still more aspiration than market

    The monetization pitch got little trust because the actual pay-per-crawl program still seems unavailable to ordinary site owners, and several comments doubted scrapers would pay enough to matter even if it launches. That turns Cloudflare’s current offer into mostly blocking and labeling, not a real replacement business model for lost search clicks. The gap between “we will help publishers get paid” and “publishers can use this today” was impossible to miss.

    Do not plan around crawler payments covering your content business any time soon. Treat crawl monetization as an experiment and continue assuming your durable choices are blocking, licensing, or finding direct audience channels.

      Attribution:
    • jerf #1
    • deadbabe #1
    • leros #1
    • zzzeek #1
  4. 04

    Bot blocking collides with accessibility and user agents

    Once access policy is framed as “human good, bot bad,” you end up blocking software that is plainly acting for a user. Comments pointed to screen-reader-adjacent tools, image description helpers, and personal agents that browse on someone’s behalf. That reframes the issue from bot identity to agency and permission. The useful line is not whether software touched the page, but whether it is acting for the site owner, the end user, or a third party extracting value at scale.

    When you set anti-bot policy, account for assistive tools and first-party agents explicitly. If your product roadmap includes agent features, assume the web will increasingly require proofs of user intent, not just browser fingerprints.

      Attribution:
    • Terr_ #1
    • jwr #1
    • akersten #1

Against the grain

  1. 01

    Google says search and training are separable

    Google’s crawler documentation claims you can block Google-Extended for training and grounding without losing inclusion or ranking in Google Search. If that holds in practice, then Cloudflare’s decision to tie multipurpose crawlers to the strictest rule is partly a Cloudflare policy choice, not an unavoidable fact of the web. It suggests some of the conflict is about enforcement leverage, not pure technical necessity.

    Check the crawler-specific controls available from each platform before assuming you must choose between search visibility and training exclusion. Cloudflare’s defaults are not the only layer where this decision can be made.

      Attribution:
    • Deathmax #1
  2. 02

    Blocking everyone but Google strengthens Google

    If publishers rely on Google for discovery and aggressively block every other crawler, they entrench the monopoly they resent. That is even more self-defeating if Google’s own results pages are keeping the clicks while smaller engines or alternative agents might still send visitors. The policy can feel defensive in the moment while making the market less competitive over time.

    Audit which crawlers actually send you traffic before you block them wholesale. Leaving room for credible alternatives may matter more now that Google is absorbing more of the answer layer itself.

      Attribution:
    • AnthonyMouse #1 #2
  3. 03

    Usage restrictions are not technically enforceable

    Once you serve a page, you cannot reliably distinguish “reading,” “indexing,” and “training” at the point of use. A recipient can store, transform, or repurpose what they fetched, and labels like robots.txt or crawler categories only work when the other side chooses to comply. That cuts against grand claims that traffic policy alone can preserve content rights on an open web.

    Use bot controls to shape traffic costs and access, not to assume downstream use restrictions will hold. If the business issue is rights or payment, you still need contracts, licensing, or legal leverage.

      Attribution:
    • sneak #1

In plain english

AI Overviews ↩
Google Search results that show an artificial intelligence generated summary above traditional links.
Anubis ↩
An open source proof-of-work challenge system used by some sites to slow down scraping bots.
CDN ↩
Content Delivery Network, a service that sits between users and websites to cache content, improve speed, and often provide security filtering.
Gemini ↩
Google’s family of artificial intelligence models and products.
Google-Extended ↩
A Google crawler control token that site owners can use to tell Google not to use their content for certain artificial intelligence uses.
Googlebot ↩
Google’s web crawler that visits pages so they can appear in Google Search and related products.
IP ↩
Intellectual property, a legal category covering rights such as copyright, patents, and trademarks.
POST ↩
An HTTP method used by browsers and apps to send data to a server, such as form submissions or API requests.
residential IPs ↩
Internet addresses assigned to home users, often used by scrapers because they look more like normal consumer traffic than datacenter addresses.
reverse DNS ↩
A network lookup that maps an IP address back to a hostname, often used to verify whether traffic really comes from a claimed service.
robots.txt ↩
A standard text file on a website that tells compliant crawlers which parts of the site they should or should not access.

Reference links

Crawler policies and verification

Anti-bot tools and examples

Related discussion examples