HN Debrief

Where did the old web go? We followed 657,607 links to find out

  • Web
  • AI
  • Infrastructure
  • Open Source
  • Developer Tools

The post comes from the author of 0.mk, a URL shortener that originally ran from 2009 to 2014 and has now been revived. Using a recovered database backup, they crawled 657,607 historical shortened links and found that 76.7 percent of the crawlable targets no longer returned a loading page. That is a generous number. Pages blocked by 403 and 429 responses may still exist, while pages that return 2xx or 3xx still count as loading even if they now land on parked domains, login walls, or removed-content notices. A big outlier from one spammy account distorted the 2011 data, but removing it did not change the broader pattern. The post also doubles as a test of whether AI can now do enough of the engineering, abuse review, spam filtering, and support work to make a tiny service like this viable again.

Treat links and community spaces as fragile infrastructure, not permanent assets. If your business depends on discoverability, archives, or user-generated content, you should plan for link rot, platform lock-in, and anti-bot friction now rather than assuming the web will preserve itself.

Discussion mood

Mostly nostalgic and pessimistic. People accepted the link-rot finding, but the stronger mood was frustration that today’s web is more centralized, more hostile to archiving and discovery, and increasingly warped by spam, anti-bot systems, and AI-generated junk.

Key insights

  1. 01

    Alive URLs overstate what survives

    A responding server is a weak test for preservation. Many links that still return HTTP now point to parked domains, login prompts, removed posts, or unrelated replacement pages, so the post’s 23.3 percent survival figure is best read as a ceiling rather than a comfortingly precise measure of what remains accessible.

    If you track content durability, measure semantic survival and not just status codes. For important references, keep snapshots or archived copies instead of trusting a live URL to mean the same thing later.

      Attribution:
    • ErigmolCt #1
  2. 02

    What vanished was ownership of community space

    The old web was not just independent-looking pages. It was communities controlling their own infrastructure, usually with a forum, mailing list, or site run by members rather than by a platform. That changes everything. When your group lives on Reddit, Facebook, Discord, or similar services, your norms, discoverability, and even your continued existence depend on a company’s policy choices and business model.

    If your community or product has strategic value, do not leave its primary home entirely on a third-party platform. Keep a domain you control and an exportable archive, even if you still use big networks for distribution.

      Attribution:
    • BrenBarn #1
    • friendzis #1
    • zem #1
  3. 03

    AI is degrading both discovery and preservation

    AI is not just adding more junk to search results. It is also pushing site owners toward heavier anti-bot defenses, login walls, and centralized protection services, which makes pages slower to access and much harder to archive or mirror. Several commenters tied this directly to the loss of the old web’s feel. The web gets worse whether AI traffic wins or site owners successfully block it.

    Expect search, crawling, and public web access to get more brittle. If your workflow depends on open indexing or cheap scraping, build fallbacks now and archive critical sources before they disappear behind defenses.

      Attribution:
    • throwaway5233 #1
    • Gormo #1
    • sebastiennight #1
  4. 04

    People miss a community, not an era

    What counts as the old web changes wildly depending on when someone first found their people online. The better frame is not a fixed technical period but the death of a specific community. A private game server, a forum, a guild, or a blog circle can die even while the software, website, or genre continues elsewhere. That makes a pure technology revival an incomplete answer.

    When evaluating community products, watch retention of relationships and norms, not just uptime or feature parity. Rebuilding the stack will not recreate the social fabric if the people and rituals are gone.

      Attribution:
    • ketzu #1
  5. 05

    Search shifted from directories to optimization warfare

    One concrete difference older users pointed to was pre-SEO discovery. Earlier web search and directories surfaced fewer, more idiosyncratic results, and pages were often written by people without trying to rank or monetize. The complaint is not that bad pages did not exist. It is that optimization became the dominant behavior, first for PageRank and now for AI systems, which changes what gets made in the first place.

    Assume discoverability systems shape supply, not just ranking. If you run content or marketplace products, the incentives of your ranking layer will determine whether creators optimize for humans or for the machine.

      Attribution:
    • sodapopcan #1
    • mkjs #1

Against the grain

  1. 01

    The old web still exists if people bother

    The strongest pushback was that much of this is selective memory. Personal sites, oddball projects, and self-hosted communities still exist, and nobody is physically prevented from registering a domain and publishing. The bigger change is demand. Many people who say they miss independent sites no longer actually visit or sustain them, while some supposedly dead formats like Diaryland are still online.

    Before blaming platforms for everything, check whether your users truly want independence badly enough to change habits. A standalone site or forum still needs repeated human attention, not just nostalgia.

  2. 02

    Modern niches can be deeper than old ones

    Another dissenting view was that today’s niche communities are often larger, faster, and more productive than their old-web predecessors. Instead of one tiny forum or one person’s homepage, you may now get large Discord servers, active video channels, live collaboration, and enough scale for creators to earn money. That can be a real upgrade, even if it comes with platform dependence.

    Do not confuse decentralization with community health. For some use cases, concentration on a modern platform produces better participation and output than a fragile standalone site ever did.

      Attribution:
    • ColdStream #1
    • AuthAuth #1 #2
  3. 03

    AI could help small services return

    A more optimistic take held that AI might not just poison the web. It could also lower the labor needed to keep modest, human-scale services alive by helping with moderation, support, spam handling, and maintenance. If mainstream users retreat into even more closed environments, that may leave room for smaller technical communities and simpler sites to thrive again.

    Cheap operational tooling can make small web products viable in categories that previously died under support and abuse costs. If you have a narrowly scoped service idea, the economics may now look different.

      Attribution:
    • MetaWhirledPeas #1
    • monk_grilla #1
    • zem #1

In plain english

2xx
A family of HTTP status codes that usually indicate a request succeeded.
3xx
A family of HTTP status codes that usually indicate a redirect to another URL.
403
An HTTP status code meaning access is forbidden, used here to describe an API error response.
429
An HTTP status code meaning too many requests were made in a short time, often used for rate limiting.
HTTP
Hypertext Transfer Protocol, the standard way web browsers and servers exchange pages, originally without encryption by default.
IRC
Internet Relay Chat, an older real-time text chat system often used in technical communities.
PageRank
A ranking algorithm that scores pages or nodes by treating links from other important pages as stronger votes.
SEO
Search engine optimization, the practice of improving a site's visibility in search engine results.
Web 2.0
A broad term for the era when the web shifted toward user-generated content, social platforms, rich browser apps, and centralized services.

Reference links

Related reading on web history and old communities

Search and indexing history

Archiving and preservation

  • Digital Data
    A comic about the practical fragility of digital preservation, cited in the conversation about lost MUD logs and vanished sites.
  • SmolNet
    Referenced in a side discussion about small, low-profile protocols such as Gopher and Gemini.

Protocols and alternative infrastructure

Critiques of URL shorteners