HN Debrief

Show HN: Getting GLM 5.2 running on my slow computer

  • AI
  • Open Source
  • Infrastructure
  • Developer Tools

Colibrì is a proof-of-concept runtime for GLM 5.2, a 744B mixture-of-experts model, built to answer a simple question: can a normal laptop run it at all. The approach is to leave the dense shared parts resident in RAM, quantized to int4, and stream only the routed experts from disk as each token activates them. That lets a machine with roughly 32 GB of memory talk to a model that would normally be far out of reach, at the cost of brutal latency. The author reports around 0.1 tokens per second on their own hardware. People still found it compelling because the point is not a fast chatbot. The point is that local inference for very large sparse models no longer requires datacenter gear if you are willing to redesign the memory hierarchy around RAM, SSD, caching, and sparse activation.

If you care about private or cheap local inference, this is a real design pattern to watch: exploit mixture-of-experts sparsity, keep a hot set in RAM, and treat fast local storage as a second-tier memory pool. Do not treat this repo as production-ready chat infra yet. Treat it as evidence that laptop-scale hardware can now host frontier-class open models if your product can tolerate asynchronous workflows and aggressive systems tuning.

Discussion mood

Strongly positive and admiring. People liked the engineering audacity and saw it as a genuine proof that sparse giant models can run on cheap hardware, even if today's throughput is too slow for normal chat and raises practical worries about SSD behavior.

Key insights

  1. 01

    Slow local models need job-style UX

    At these speeds the right interface stops being interactive chat and starts looking like a ticket system or overnight worker. That reframes the project from a disappointing chatbot into a plausible local batch engine for coding, document work, or long-running agent jobs where privacy and cost matter more than latency.

    If you want to productize this class of runtime, design around queued tasks, resumable jobs, and delayed results. Do not benchmark success only by chat responsiveness.

      Attribution:
    • codazoda #1
    • walrus01 #1
  2. 02

    Linux mmap can replace bespoke residency logic

    A llama.cpp experimenter said they got to a similar place by memory-mapping the full model and letting the Linux kernel decide what stays hot, instead of hand-managing residency. They also paired it with Medusa for speculative decoding. That matters because it suggests part of Colibrì's novelty may be portable into mainstream inference stacks rather than requiring a custom runner forever.

    Watch for these ideas to land in llama.cpp or similar engines. If they do, the barrier to testing SSD-streamed giant models drops sharply because teams can iterate inside familiar tooling.

      Attribution:
    • Cieric #1 #2
  3. 03

    SSD wear is mostly a swap problem

    The useful correction was that reads are not what kill the drive here. Trouble comes from memory pressure that pushes other pages or caches to disk. Several comments pointed to concrete mitigations like keeping the KV cache in RAM, measuring actual disk writes, disabling swap, or lowering swappiness before declaring the approach unsafe. That turns a vague hardware fear into an observable operating-system problem.

    Before rejecting disk-streamed inference on reliability grounds, instrument write amplification on your target machine. Then tune swap and memory layout based on numbers, not on the assumption that heavy reads automatically wear out SSDs.

      Attribution:
    • voidmain0001 #1
    • killerstorm #1
    • Archit3ch #1
    • gcr #1
    • throwa356262 #1
  4. 04

    Weight streaming is becoming a general inference pattern

    People independently described the same architecture across Apple Silicon LLM work, DiffusionGemma on Metal, and image or video generation under strict VRAM budgets. The common lesson is that staged loading, compression, and LRU caching are turning storage into an extension of memory across both language and diffusion models. This is starting to look less like a one-off stunt and more like an emerging systems layer that should be upstreamed into shared runtimes.

    Expect future inference frameworks to expose explicit RAM, VRAM, and storage budgets instead of assuming full model residency. Teams building local AI products should plan for multi-tier memory management as a first-class feature.

      Attribution:
    • Archit3ch #1 #2
    • mmastrac #1
    • kodablah #1

Against the grain

  1. 01

    Free hosted GLM is more practical now

    For actual use today, free cloud access to GLM 5.2 beats waiting a minute per token on a laptop. That cuts through some of the excitement. The local setup wins on autonomy and experimentation, not on throughput or convenience yet.

    If your goal is shipping a user-facing feature soon, compare this against hosted GLM before investing engineering time. Local giant-model runtimes make more sense when privacy, offline use, or control outrank speed.

      Attribution:
    • charcircuit #1
  2. 02

    Better hardware paths may beat SSD streaming

    Some commenters pointed to other routes that may deliver higher real-world performance, including Mac-focused projects like flash-moe and hardware like Intel Optane that was designed for this memory-storage middle ground. That weakens any claim that Colibrì's exact architecture is the inevitable solution. It is one promising workaround in a hardware landscape that still has better-fit options when available.

    Treat the technique as hardware-contingent. Benchmark against unified-memory Macs, larger RAM boxes, and alternative storage tiers before committing to SSD-first designs.

      Attribution:
    • bahmboo #1
    • xtracto #1

In plain english

GLM 5.2 ↩
A recent language model from Z.ai that commenters used as a price and capability comparison point.
int4 ↩
A very low-precision numeric format using 4 bits per value, often used to compress model weights for inference.
iostat ↩
A system tool that reports CPU and disk input-output statistics.
KV cache ↩
Key-value cache, a memory structure used during transformer inference to avoid recomputing earlier tokens and to support long context windows more efficiently.
LLM ↩
Large language model, an artificial intelligence system trained on large text datasets to generate and analyze language.
Medusa ↩
A speculative decoding method for language models that adds extra prediction heads to speed up generation.
Metal ↩
Apple's graphics and compute programming framework used to run GPU workloads on macOS and iOS devices.
Mixture-of-Experts ↩
A model architecture where only a subset of specialized sub-networks, called experts, is activated for each token instead of the whole model.
MoE ↩
Mixture of Experts, a model architecture that activates only a subset of its parameters for each token instead of using the full network every time.
Optane ↩
Intel Optane, a discontinued class of memory and storage products designed to sit between RAM and SSDs in speed and latency.
swappiness ↩
A Linux kernel setting that controls how aggressively the system moves memory pages to swap.
vmstat ↩
A system tool that reports memory, process, paging, and system performance statistics.
VRAM ↩
Video RAM, memory attached to a GPU and used for graphics or AI model computation.

Reference links

Related inference engines and projects

  • fastllm
    Mentioned as a possibly similar project for local model inference.
  • thinfer
    A related project for image and video generation that streams weights under a VRAM budget.
  • flash-moe
    Shared as another project that runs very large mixture-of-experts models on Apple hardware.
  • hypura
    Pointed to as a related and possibly more general-purpose system.

Papers and technical methods

Operational tuning references