HN Debrief

I built a 500k-domain search engine for makers in a weekend for $10

  • Search
  • AI
  • Open Source
  • Developer Tools

The post describes a weekend project called Marlin, a lightweight search system for makers that crawls domains, skips empty pages and bot walls, uses a small local language model to generate short descriptions and tags, and keeps the metadata footprint tiny enough that roughly 500,000 domains fit in about 1 GB. The code is already public. The author was explicit in the comments that this is not meant to compete with Google, Bing, DuckDuckGo, or ChatGPT. The point is to fork it, define your own crawl policy, start from a seed list of sites you care about, and build a personal or niche search engine around that slice of the web.

If you care about search or discovery in a niche market, the opportunity is not another general web search box. It is tooling that lets users seed, crawl, classify, and search a domain-specific corpus they control.

Discussion mood

Mixed but interested. People liked the idea of personal or niche search and agreed web discovery is underserved, but many were irritated by the AI-generated write-up and underwhelmed by the framing of 500,000 domains as a technical feat.

Key insights

  1. 01

    Search is now a good niche tool

    A narrow search engine can be worth building even when it is nowhere near web-scale. The leverage is not broad coverage. It is better treatment of a specific corner of the internet. Comments tied this to the current state of discovery, where mainstream search is weak enough that an auto-labeled directory for a chosen niche starts to look practical. That makes projects like this a product pattern, not just a weekend hack.

    Look for markets where users would prefer a search tool over a feed, but only if it is tuned to their exact corpus. A smaller, opinionated index can beat a general engine when the job is discovery inside a niche.

      Attribution:
    • marginalia_nu #1 #2
    • jeromechoo #1
  2. 02

    Local models are shifting search back on-device

    Cheap local inference changes what counts as a personal computing task. Translation, labeling, summarization, and lightweight classification used to require hosted APIs. Now they can sit beside a local crawl and produce a usable custom index. The comments framed this as the familiar pattern where new capabilities start in the cloud, then move onto local machines once models and hardware get efficient enough.

    If your product still assumes every intelligence feature must call a hosted model, revisit that assumption. Local-first versions can cut cost, reduce lock-in, and make custom data workflows viable for individuals and small teams.

      Attribution:
    • eggbrain #1
    • an0malous #1
    • dylan604 #1
  3. 03

    Indexing is solved but ranking is not

    Throwing 500,000 documents into Lucene is easy. That is not the hard part. The hard part is avoiding the dead, noisy search experience people associate with internal tools like Confluence. The value in a project like this comes from crawl policy, labeling, and retrieval choices that shape the corpus before the query ever runs.

    Do not confuse standing up search infrastructure with delivering good search. Put product effort into corpus selection, metadata quality, and ranking signals, because that is where users feel the difference.

      Attribution:
    • pimlottc #1
    • marginalia_nu #1
  4. 04

    The corpus is biased by bot defenses

    This system gets affordability by walking around the messiest parts of the modern web. It skips JavaScript-heavy pages, captcha walls, Cloudflare protection, and parked or empty sites. That keeps the pipeline simple, but it also means the resulting index overrepresents simpler, more open websites and underrepresents much of the commercial web.

    Treat results from a crawler like this as a deliberate slice of the web, not a neutral sample. If you build on it, be explicit about what classes of sites are missing and whether that is a feature or a problem.

      Attribution:
    • sandeepkd #1
    • voidUpdate #1
    • dreamforever #1
  5. 05

    Domain sourcing is the real plumbing

    Getting a useful seed set is a bigger practical issue than the post first made clear. Comments pointed to certificate transparency logs, ICANN zone files, Common Crawl, and Derek Sivers' guide to downloading TLD data. The author later clarified that users must bring their own starting list, then let the spider expand from there. That shifts the challenge from search UI to acquisition strategy.

    If you want to turn this pattern into a product, invest early in repeatable corpus assembly. The quality and legality of your seed data will matter more than the query box.

      Attribution:
    • alightsoul #1 #2
    • frogger8 #1
    • tpowell #1
    • eggbrain #1

Against the grain

  1. 01

    The utility is still fuzzy

    Even after the explanation, some readers did not see why this beats existing search tools for real tasks. If the end user mostly wants to find information fast, a custom crawl plus local indexing can feel like a complicated way to recreate what Google or DuckDuckGo already do well enough.

    Before building niche search, define the user job in operational terms. If you cannot name the queries that fail in mainstream tools and succeed in your corpus, the product may remain a hobby demo.

      Attribution:
    • fg137 #1
    • prepend #1
  2. 02

    500k domains is not an impressive scale

    The numbers did not awe everyone. Comments pointed out that even AltaVista ran on hardware that now looks tiny, and that modern laptops can hold search indexes that once needed serious infrastructure. That reframes the post from a scaling story to a packaging story. The novelty is cheap assembly with local models, not the raw document count.

    Do not pitch projects like this on scale alone. Users and investors will care more about what the system surfaces, filters, or enables than about a domain count that modern hardware handles easily.

      Attribution:
    • eichin #1 #2
    • coredog64 #1
  3. 03

    AI-written documentation destroys trust

    The harshest criticism landed on the article itself. Readers were not objecting to AI as a tool. They objected to unedited, bloated prose that made them do the filtering work. Several comments went further and argued that when the explanation is machine-generated and lightly checked, the burden of verification shifts to the reader. That makes technical claims feel less trustworthy, not more accessible.

    If you use AI to draft launch posts or docs, do the human editing work in public-facing material. Clear writing is part of the proof that you understand what you built.

      Attribution:
    • headz #1
    • criddell #1
    • augment_me #1
    • order-matters #1
    • uean #1

In plain english

CAPTCHA ↩
A test used on websites to distinguish humans from bots, often by asking users to solve a visual or logic challenge.
Cloudflare ↩
A company that provides CDN, caching, and network services to speed up and protect websites.
Confluence ↩
Atlassian's workplace wiki and documentation product, often cited for mediocre built-in search.
ICANN ↩
Internet Corporation for Assigned Names and Numbers, the organization that coordinates domain names and related internet identifiers.
JavaScript ↩
A widely used programming language for web development and increasingly for servers, scripts, and tooling.
Kagi Small Web ↩
A feature from the search engine Kagi that focuses results on smaller independent websites.
Lucene ↩
A widely used open source search library that powers many full-text search systems.
TLD ↩
Top-level domain, the last part of a domain name such as `.com`, `.eu`, or `.net`.

Reference links

Project and code

Domain and crawl data sources

Search history and related products