HN Debrief

What happened to TheNumbers.com

  • AI
  • Infrastructure
  • Economics
  • Open Source
  • Security

The post is a warning built around TheNumbers.com, a long-running film industry site whose public pages recently returned in a reduced form after going offline. It suggests several forces collided at once: AI-era scraping that makes free public pages expensive to serve, an aging codebase that became risky to expose, and a stronger incentive to probe for weaknesses because prediction markets can reward anyone who gets box office data a little early. That last point matters because The Numbers is not just a blog or hobby archive. It is a reference source people use to settle bets and inform paid products, so even a small timing edge can be worth money.

If you run a public data or content site, assume abusive automated access is now a baseline operating cost and design around it before traffic spikes. Also separate your threat models: scraping, infrastructure economics, and adversarial users motivated by trading or gambling need different defenses and different business decisions.

Discussion mood

Frustrated and uneasy. People broadly accepted that abusive scraping is raising costs and pushing maintainers off the open web, but they were skeptical of treating this single site as a clean AI story because legacy architecture and prediction-market incentives likely mattered just as much.

Key insights

  1. 01

    Search endpoints are the real cost sink

    The expensive part was not ordinary indexing. One operator said Google-level crawling was manageable and even useful, while a single scraper hitting nearly every combination of 10 to 12 search facets drove more than 100 times the traffic and blew through cache. That changes the lesson from “bots are bad” to “parameterized public interfaces are now a liability unless you tightly constrain them.”

    Audit every public endpoint that can generate combinatorial query explosions. Precompute popular views, cap parameter spaces, and put hard limits in front of search-like routes before one crawler turns a cheap site into a bandwidth bill.

      Attribution:
    • primitivesuave #1 #2 #3
  2. 02

    Bulk downloads do not stop wasteful scraping

    Making the full dataset available as a ZIP sounds like the obvious answer, but commenters pointed out that Wikipedia and Read the Docs already offer bulk access and still get hammered. The problem is not lack of sanctioned access. It is that many crawlers are badly built, indifferent to cost, or chasing HTML specifically, so they keep pulling pages anyway.

    Offer bulk exports and APIs for good actors, but do not expect them to reduce hostile or careless crawl load. You still need controls at the edge for HTML delivery.

      Attribution:
    • jjgreen #1
    • codemonkey-zeta #1
    • antisthenes #1
  3. 03

    Robots.txt only works on polite bots

    Blocking major AI crawlers in robots.txt helps at the margin, but people described plenty of leakage. Some bots still behave inefficiently even when identified, and much of today's automated traffic arrives through headless Chrome and residential proxies that are hard to distinguish from humans. That means the old “declare your preferences and move on” model is no longer enough.

    Treat robots.txt as a courtesy layer, not a defense layer. Monitor actual request patterns and be ready to gate expensive routes with rate limits, challenges, or authentication.

      Attribution:
    • pixelesque #1
    • frereubu #1
    • spiderfarmer #1
  4. 04

    Technical hardening does not restore the business model

    Several people pushed back on the easy “just rewrite it as static” answer. Even if a CDN and cache can keep the site online, that does not solve the core asymmetry where bots extract the data and reduce reasons for humans to visit, subscribe, or click through. For sites that depend on scarcity, attribution, or direct audience relationships, surviving the traffic is not the same as preserving the product.

    When planning anti-bot work, separate uptime from economic viability. A cheaper architecture is useful, but you may also need licensing, delayed public release, gated access, or a different value proposition.

      Attribution:
    • jaredwiener #1 #2
    • tptacek #1
    • freediddy #1
  5. 05

    Prediction markets create an attack incentive

    The strongest non-AI point was that early access to a trusted data source can be monetized through betting. That turns a niche reference site into infrastructure for speculation and gives both scrapers and intruders a reason to probe for timing advantages. In that frame, the site was not just overloaded. It became a target because the data could move money.

    If your data can settle trades, influence prices, or trigger payouts, model attackers as financially motivated adversaries. That calls for stronger isolation, delayed publication workflows, and tighter operational security than a normal public content site.

      Attribution:
    • abetusk #1
    • shermantanktop #1
    • Terr_ #1
  6. 06

    Cloud economics made an old web problem worse

    People noted that crawling and cache design were not new problems. The newer pain comes from serving public content on per-gigabyte cloud pricing while modern apps are often slower and more dynamic than the PHP-era sites they replaced. In other words, bad bot behavior is colliding with architectures and hosting models that make each miss more expensive.

    Revisit whether your public-facing workload belongs on metered cloud egress at all. For high-volume public data, static hosting, cheaper bandwidth providers, or CDN-heavy designs can matter as much as application code.

      Attribution:
    • ethagnawl #1
    • zackmorris #1
    • PunchyHamster #1

Against the grain

  1. 01

    This is mostly a traffic engineering failure

    One blunt response argued that if public demand can take a site down, the operator failed to design for abuse. The point is harsh, but it reframes some of the problem as predictable systems work rather than a uniquely AI-era shock. Public endpoints that can be enumerated will be enumerated.

    Do not rely on user goodwill when exposing queryable public surfaces. If a route can be abused mechanically, assume it will be and budget engineering time accordingly.

      Attribution:
    • x3haloed #1
  2. 02

    Prediction markets matter more than AI here

    A recurring dissent was that the article overstates AI and understates the gambling angle. In this view, a creaky 30-year-old site serving market-relevant data was always going to draw attackers once real money depended on being slightly earlier than everyone else. AI scraping is a multiplier, not the root cause in this case.

    When diagnosing incidents, rank incentives before technologies. If money can be made from getting data early, removing or regulating that incentive may reduce pressure more than another round of bot mitigation.

      Attribution:
    • nater5000 #1
    • jambalaya8 #1
    • tehjoker #1
  3. 03

    Not everyone accepts AI's value bargain

    A few comments rejected the idea that better models compensate creators for having their work absorbed into commercial systems. They were not arguing about infrastructure at all. They were saying the social contract changed. The pleasure of making things for people does not feel interchangeable with feeding a model that captures the relationship and the revenue.

    If your strategy depends on voluntary public contributions, assume some creators will pull back even if scraping becomes technically manageable. Trust and perceived fairness now shape supply as much as tooling does.

      Attribution:
    • podgietaru #1 #2
    • jtuple #1

In plain english

AI ↩
Artificial intelligence, here mainly referring to software systems and crawlers associated with large language models.
CDN ↩
Content Delivery Network, a service that caches and serves content from locations closer to users to reduce load on the origin server.
headless Chrome ↩
A version of the Chrome browser that runs without a visible user interface and is often used for automation or scraping.
residential IPs ↩
Internet addresses assigned to home users that can make automated traffic look like normal consumer browsing.
robots.txt ↩
A standard text file on a website that tells compliant web crawlers which pages or paths they should avoid.

Reference links

Cloud data access and publishing

Caching and web architecture

  • Magnet URI scheme
    Mentioned during a tangent about peer-to-peer content distribution and content-addressable web delivery.
  • Swytch paper on caching architecture
    Shared as relevant prior work on difficult cache invalidation and architecture problems.
  • Swytch
    Linked alongside the paper as an implementation related to the caching discussion.

Crawler controls and bot economics

Related Hacker News discussion