The post is a year-long field report from the owner of PatronView, a 1.5 million-page site that tracks charitable foundations and grants. The claim is simple: bot traffic, especially from AI crawlers and low-quality scrapers, now dominates the site, pushes hosting bills far above normal, poisons analytics, and turns basic site operation into an anti-abuse exercise. The author walks through the progression from odd traffic spikes to sustained crawling, then through the mitigations that helped, mostly Cloudflare rules, geoblocking, and route-specific defenses. A key datapoint that stuck with people was the mismatch between scraping and actual value sent back. One commenter running a financial data site saw Claude fetch about 205,000 pages in 72 hours and send one referral. That fit the post’s core complaint that AI companies are extracting value while pushing bandwidth, compute, and operational costs onto everyone else.
The most useful consensus was less ideological and more practical. Static sites on cheap VPSs are still fine. The pain shows up when pages are dynamic, hit databases, expose expensive routes like search, logs, history views, or unbounded calendars, or sit on usage-based cloud products where a scraper becomes a surprise bill. Several operators said the fix is often boring engineering before fancy bot detection. Cache aggressively, precompute what you can, put hard bounds on date and archive navigation, and do not expose expensive operations to unauthenticated crawlers. Others added that many bots are not even smart enough to need sophisticated defenses. Simple JavaScript cookie checks, fetch-header checks, or lightweight challenge pages keep out a large share because the attackers optimize for cheap mass scraping, not perfect browser simulation.
Where the conversation sharpened was on the tradeoff. People are increasingly using LLMs, scripts,
RSS readers, translation tools, and other agents as their way of reading the web, so blocking all non-browser traffic can lock out legitimate users and future discovery paths. But most commenters still landed on a blunt economic point: site owners do not owe free compute to every agent that claims to act for a user, especially when those agents revisit the same pages constantly and rarely send traffic back. Cloudflare and
Anubis were treated as imperfect stopgaps, not solutions. Cloudflare catches a lot, but many people described false positives, broken access for old browsers, VPN users, or no-JavaScript users, and discomfort with one company quietly deciding who gets through. Anubis got praise as a cheap deterrent for hobby sites, then got torn down as easy to bypass with native code or
GPU solvers if scrapers ever care enough. The practical conclusion was grim but clear: the current web stack has no clean mechanism for machine access that preserves openness, monetization, and operational sanity at the same time, so operators are improvising with friction and hoping to stay on the right side of the false-positive curve.
patronview.com
SignalBloom traffic screenshot