HN Debrief

Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

  • AI
  • Open Source
  • Developer Tools
  • Infrastructure

WASTE is a custom inference engine for Moonshot's Kimi K3 that keeps RAM use low by storing most of the model on disk and streaming the needed weights as it runs. The headline claim is 29 GB of RAM for a 4k context window on Apple Silicon, with output around 0.5 tokens per second. The project also uses a heavily compressed representation rather than the untouched full model, which confused people because the README initially sounded more like "native precision" than "requantized experiment." Once that was clear, the project read less like a practical alternative to cloud inference and more like an impressive demo of how far storage-backed local inference can be pushed.

Treat this as a boundary-pushing systems experiment, not a practical local AI stack. If you care about local inference, the useful question is whether your workload can run unattended overnight and whether SSD streaming plus aggressive quantization is worth the quality, power, and complexity tradeoff versus paying for hosted models or using smaller local ones.

Discussion mood

Mixed leaning skeptical. People liked the systems hack and the proof that a huge model can be squeezed onto modest RAM, but the dominant reaction was that 0.5 tokens per second is too slow, too power-inefficient, and too compromised by quantization to beat hosted models for most real work.

Key insights

  1. 01

    Manual paging can beat mmap hard

    Manual prefetch and cache control are the real systems idea here. A commenter tied it to database engines that bypass generic kernel paging because they know the access pattern in advance. For a model that streams predictable expert weights every token, that can outperform vanilla memory mapping enough to justify a custom runtime at all.

    If you are exploring SSD-backed inference, do not assume the operating system page cache is close to optimal. Profile explicit prefetch, residency control, and pipeline overlap before deciding the idea is dead.

      Attribution:
    • jnwatson #1
    • mappu #1
  2. 02

    This is not untouched Kimi K3

    The memory win comes with a model-quality trade. A Deltafin developer said WASTE is using a 3-bit residual requantization rather than the full unaltered K3, and estimated it cuts expert traffic by about 34 percent. That makes the project easier to understand: it is not just clever streaming, it is clever streaming plus aggressive compression.

    When you compare "runs giant model locally" projects, separate the storage trick from the model-fidelity trick. Speed, memory, and quality are being traded at the same time.

      Attribution:
    • chrisringrose #1
  3. 03

    Overnight workflows are the only sane fit

    The viable use case is not chat. It is unattended work that can finish before a deadline, like overnight code review, project analysis, or other long-running background jobs. That reframes the benchmark from "tokens per second" to "did the result land by morning," which is the only lens under which 0.5 tokens per second sounds remotely useful.

    If you are evaluating very slow local models, test them inside deadline-based batch workflows. If you need back-and-forth iteration, move on immediately.

      Attribution:
    • walrus01 #1
    • Lerc #1
    • cadamsdotcom #1
  4. 04

    Power economics still favor datacenters

    The rough math was brutal. At around 42 watts sustained and 0.5 tokens per second, commenters estimated roughly $5 per million tokens in electricity alone. Another commenter pointed out that modern GPU clusters are on the order of 1000 to 2000 times more power efficient for this workload, because SSD streaming and low-bandwidth host memory are the wrong substrate for giant model inference.

    Do not confuse local feasibility with local efficiency. If your goal is low-cost volume inference, consumer SSD plus DRAM setups are still the wrong baseline.

      Attribution:
    • bgirard #1
    • AnotherGoodName #1
    • herf #1
  5. 05

    Bandwidth is the real bottleneck

    Storage speed is not the only limiter. One commenter argued the ceiling is memory bandwidth into the CPU, since each generated token drags tens of gigabytes of traffic and DDR5 bandwidth is finite before compute even starts. That also explains why the author's attempt to use Metal on macOS underperformed NEON for this specific design. The workload is bottlenecked by moving bytes around more than by raw accelerator throughput.

    When you design offloaded inference systems, treat bandwidth accounting as first-order architecture work. Faster SSDs alone will not save a pipeline that is already starved on DRAM and sequential compute.

      Attribution:
    • marcobambini #1
    • withinboredom #1
  6. 06

    Personal writing corpus as style control

    One concrete use of LLMs stood out from the README fight. A commenter described building a local SQLite archive of years of posts, commit messages, and discussions, then letting Claude query it to mimic their writing style and analyze their own history. That turns "write in my voice" from a hand-wave into a real retrieval setup, and it points to a better way to generate docs than accepting default Claude prose.

    If you use LLMs for outward-facing writing, build a retrieval corpus from your own past work and make the model pull from it. Style transfer works better when it is grounded in actual examples instead of one prompt line.

      Attribution:
    • matheusmoreira #1 #2

Against the grain

  1. 01

    The value is setting a new frontier

    The strongest defense was that practicality today is beside the point. Showing that a huge model can run in 29 GB of RAM, however slowly, expands the design space for local inference. The interesting part is not this exact speed but the possibility that future compression and hardware improvements stack on top of the same idea.

    Do not dismiss systems work just because the first version is unusable for production. Frontier demos often matter because they reveal a path, not because they are ready now.

      Attribution:
    • ruler88 #1
    • fsuts #1
  2. 02

    Some hosted pricing is not obviously better

    One commenter pushed back on the blanket claim that cloud always wins on cost by pointing to GitHub Copilot credit pricing for Opus, where the effective token cost can be worse than the rough electricity estimate here. That does not rescue WASTE as a general solution, but it does weaken the lazy assumption that every paid model is automatically cheaper.

    Compare against the actual SKU you would buy, not an abstract idea of "the cloud." Token pricing varies enough that niche local setups can occasionally make economic sense for specific products or plans.

      Attribution:
    • Austiiiiii #1
  3. 03

    The README problem is fixable

    Not everyone took the Claude-written README as evidence the project itself was sloppy. One commenter said the document was at least scannable and that the useful criticism was concrete missing information about quantization, quality loss, and hardware tradeoffs. In other words, the issue was bad documentation choices, not proof that the underlying code is worthless.

    When evaluating LLM-assisted projects, separate presentational slop from technical substance. Ask for missing benchmark and model details before concluding the implementation is junk.

      Attribution:
    • yearolinuxdsktp #1

In plain english

3-bit residual ↩
A very low-precision quantization scheme that stores values using roughly three bits plus extra correction information.
4k context ↩
A context window of about 4,000 tokens, meaning how much input text the model can consider at once.
Apple Silicon ↩
Apple's ARM-based processors used in modern Macs.
datacenter GPUs ↩
High-end graphics processors in server racks that are optimized for large-scale AI inference and training.
DDR5 ↩
A recent generation of main memory with higher bandwidth than older DDR standards.
expert traffic ↩
The amount of model-weight data that must be moved to run the selected experts in a mixture-of-experts model.
GGUF ↩
A file format commonly used to package language models for local inference tools such as llama.cpp.
Kimi K3 ↩
A very large open-weight language model from Moonshot AI that is expensive to run because of its size.
llama.cpp ↩
A popular open source C and C++ project for running language models locally on CPUs and GPUs.
memory-mapping ↩
A way for software to access a file as if it were memory, letting the operating system load needed pieces on demand.
Metal ↩
Apple's graphics and compute programming framework for GPUs.
NEON ↩
ARM's SIMD instruction set for speeding up data-parallel work on CPUs.
prefetch ↩
Loading data before it is needed so computation does not stall waiting for storage or memory.
quantization ↩
Compressing model weights into lower precision numbers so the model uses less memory and runs with less bandwidth.
requantization ↩
Converting model weights into a lower-precision representation to reduce memory and bandwidth needs.
SQLite ↩
A lightweight database stored in a single file, often used for local apps without a separate database server.
SSD ↩
Solid-state drive, a storage device that uses flash memory instead of spinning disks.
tokens per second ↩
A model speed measure showing how many chunks of text the model can generate each second.

Reference links

Related inference projects

  • Deltafin
    Direct comparison point for running Kimi K3 from storage, discussed as using the unaltered full model rather than WASTE's requantized version.
  • Colibri
    Another project mentioned as a similar approach for running a very large model with storage-backed techniques.

Author and project references

  • Gravity programming language
    Linked by the author to show prior hand-written software work in response to criticism that the project looked LLM-generated.
  • Deslopper
    A tool shared as a way to strip stereotypical LLM prose from generated documentation in continuous integration.

Background discussion