HN Debrief

Running a 28.9M parameter LLM on an $8 microcontroller

  • AI
  • Hardware
  • Open Source
  • Infrastructure

The project is a demo of running a 28.9M parameter LLM on an ESP32-S3, a small microcontroller with external PSRAM and flash, by squeezing the model hard enough that it can stream weights and still produce roughly 9.7 tokens per second. That got people excited not because anyone wants a smart toaster chatbot, but because it shows how far the floor has moved for embedded inference. Several comments pointed out that tiny text-to-speech and speech-to-text models are already in this size range, which makes fully offline voice interfaces on very cheap hardware feel much closer than they did a year ago.

Treat this less as a cheap chatbot and more as proof that embedded AI is moving from toy demos to real offline edge use cases. If you build hardware products, start watching what tiny speech, vision, and control models can do locally before assuming you need cloud inference or a full Linux-class board.

Discussion mood

Strongly positive and impressed. People loved the engineering stunt, the price-performance of current embedded hardware, and the hint that offline edge speech and other small models are becoming practical, while staying realistic that this does not generalize cleanly to large-model inference.

Key insights

  1. 01

    Why this trick does not scale cleanly

    What makes the ESP32 demo fast enough is not just storing weights in cheap flash. It is the combination of a model trained to use per-layer embeddings and a storage path where flash is memory-mapped with low overhead. That falls apart on desktop-style SSD streaming, where NVMe command handling and much higher latency dominate. The result is what recent GLM-5.2 experiments showed: technically possible, but often measured in 10 to 20 seconds per token.

    Do not treat this as evidence that storage bandwidth problems for big models are solved. If you want edge inference on constrained devices, co-design the model and memory layout instead of assuming a larger model can be streamed from slower media.

      Attribution:
    • chrishynes #1
    • AussieWog93 #1
    • Rohansi #1
    • monocasa #1
    • kamranjon #1
  2. 02

    Small voice models are the near-term payoff

    The most believable product angle here is not a miniature chatbot. It is offline speech. Comments noted that text-to-speech and speech-to-text models are already landing in roughly the 20M to 30M parameter range, which puts always-on local voice I/O on hardware this cheap within reach. That is a much cleaner fit for the latency, privacy, and power envelope than open-ended language generation.

    If you ship consumer or industrial devices, evaluate local voice pipelines now. Cheap hardware that can hear and speak without a network connection is turning into a real design option.

      Attribution:
    • helsinkiandrew #1
    • rao-v #1
  3. 03

    The bigger story is embedded hardware progress

    Several comments used this project as a marker for how much embedded compute has improved. Boards like Milk-V Duo now bundle Linux-capable cores, RISC-V vector support, real-time cores, and even small TPUs at prices that used to buy a basic microcontroller. That changes the software story too, because these chips increasingly support mainstream toolchains and higher-level languages instead of forcing everything into bare-metal C.

    Revisit old assumptions about what class of chip your product needs. Some workloads that once required a Raspberry Pi or custom accelerator may now fit on much cheaper embedded parts with simpler power and deployment tradeoffs.

      Attribution:
    • titzer #1 #2
    • brucehoult #1
    • pjmlp #1
  4. 04

    Impractical demos still move the field

    The pushback that this is useless missed the point. Comments tied this kind of stunt to the same pattern that made MCU computer vision normal: somebody first does the obviously awkward version, learns the constraints, and turns that into reusable techniques. Even when the exact demo never ships, the engineering work around quantization, memory layout, and pipelining tends to survive.

    Do not dismiss extreme constraint experiments inside your team as pure novelty. They are often the fastest way to discover optimizations that later matter in real products.

      Attribution:
    • skippyfish #1
    • dannyw #1 #2
    • thomascountz #1
    • rufasterisco #1

Against the grain

  1. 01

    Microcontroller LLMs may stay a dead end

    The sharpest skeptical view was that this environment is so constrained that language models on MCUs are more curiosity than platform. In that framing, the demo proves grit, not usefulness, because there is too little memory and too little headroom for models large enough to be broadly valuable.

    Keep your use-case filter on. If your product needs richer reasoning or flexible dialogue, skip the heroics and move up to hardware that gives you enough memory and thermal budget to matter.

      Attribution:
    • skippyfish #1 #2
  2. 02

    Training work is more interesting than inference stunt

    One comment argued the real achievement sits upstream in the modeling, not the deployment target. In the very small parameter regime, every bit of benchmark performance forces researchers to remove waste and redundancy at the model level. That makes tiny models a useful pressure test for whether current training methods are still carrying dead weight.

    Watch for advances in compact model training, not just clever inference hacks. Better small-model quality will unlock more products than squeezing the same mediocre model onto smaller chips.

      Attribution:
    • DoctorOetker #1

In plain english

ESP32-S3 ↩
A low-cost microcontroller from Espressif that includes wireless features and can connect to external memory for embedded projects.
GLM-5.2 ↩
A family of language models referenced in the comments as an example used in SSD-streaming experiments.
LLM ↩
Large language model, a type of AI model trained to predict and generate text and code.
MCU ↩
Microcontroller unit, a small self-contained computer on a chip used in embedded devices.
NVMe ↩
Non-Volatile Memory Express, a high-speed interface commonly used by solid-state drives.
PSRAM ↩
Pseudo-static random-access memory, a cheaper type of memory often used as expandable RAM in embedded devices.
quantization ↩
A technique that reduces the precision of model weights and activations so a model uses less memory and compute.
RISC-V ↩
An open instruction set architecture that chip designers can implement without paying licensing fees.
SRAM ↩
Static random-access memory, a fast type of memory often built into chips for low-latency access.

Reference links

Project and hardware references

Related edge AI projects

  • Espressif esp-vision
    Given as evidence that MCU computer vision moved from impractical experiment to working product category.
  • AI on the Edge Device
    Shared as a practical ESP32 plus camera project that reads utility meters locally with computer vision.

Scaling and performance references

Microcontroller economics and flash behavior