HN Debrief

Run Qwen3.8 27B locally: real numbers from my Mac Studio

  • AI
  • Hardware
  • Developer Tools
  • Privacy

The post walks through running Qwen 3.8 27B locally on a Mac Studio and shares real throughput numbers for different setups, with the headline result landing around the mid-teens of tokens per second. That made the piece look like a cautionary tale about expensive Apple hardware underdelivering for local LLM use.

Do not treat a single local LLM benchmark as a hardware verdict unless it names the runtime, quantization, speculative decoding setup, and context-length behavior. If you are evaluating Macs versus Nvidia or AMD for local AI, the bigger decision is software maturity and memory footprint, not just raw listed token speed.

Discussion mood

Frustrated and skeptical. People mostly thought the posted numbers were misleading because the setup looked misconfigured or poorly optimized, and they used that as evidence that local LLM tooling is still too fiddly for something that should be a straightforward benchmark.

Key insights

  1. 01

    The slowdown likely comes from missing MTP

    Speculative decoding via MTP looks like the most plausible reason Qwen 3.8 appeared half as fast as Qwen 3.6. Several people pointed out that the two versions should not differ much at equal size and quantization, so a missing or broken MTP path, a bad quant export, or runtime support that has not caught up is a better explanation than "3.8 is slower." That reframes the post from a model benchmark into a software stack benchmark.

    If you benchmark local models, log whether MTP or speculative decoding is active and which model artifact actually contains the right heads. Without that, you can easily publish a runtime bug as if it were a hardware result.

      Attribution:
    • woadwarrior01 #1
    • kgeist #1
    • kennywinker #1
    • Infernal #1
  2. 02

    Context length changes the speed story

    Raw tokens per second at an empty prompt hides what happens once real context loads up. People reported setups that start around 40 tok/s and then collapse toward 10 tok/s as context grows, while others shared better numbers across 0 to 16k context with different quantizations and speculative decoding enabled. The practical performance question is not peak decode speed. It is how fast the model stays once you are doing actual coding or retrieval work with nontrivial context.

    Ask for throughput curves by context window, not a single speed number. For product work, long-context degradation will matter more than the best-case screenshot.

      Attribution:
    • seanmcdirmid #1 #2
    • hypersoar #1
    • petercooper #1
  3. 03

    Prefill matters more for coding agents

    For agentic coding and retrieval-heavy workflows, decode speed is only half the job. One commenter called 14 tok/s decode merely "okayish" and said the real problem was 93 tok/s prefill, because agents spend a lot of time reading large prompts, tools, and prior context before they emit anything. That matches reports from Nvidia users getting dramatically higher prefill and running multiple simultaneous agents with long windows, which is why the same model can feel sluggish on one box and productive on another.

    When comparing local inference hardware, track prefill throughput separately from generation throughput. If your workload is coding agents or large-context RAG, optimize for prompt ingestion first.

      Attribution:
    • arjie #1
    • kgeist #1
    • oceanplexian #1
    • spwa4 #1
  4. 04

    The missing product is a true local LLM appliance

    What people want is not another inference engine. They want a Mac-native app that picks the right model, quant, and settings automatically and gives predictable performance with minimal setup. The comparison to Draw Things made the gap obvious. Image generation has a mainstream local UX on Mac. Local LLMs still require too much switching between Ollama, LM Studio, llama.cpp, oMLX, model formats, and harnesses just to get a sane result.

    There is still room for a strong local AI product that wins on setup and defaults rather than model access alone. If you are building in this space, the wedge is operational simplicity, not one more backend.

      Attribution:
    • liuliu #1 #2
    • __mharrison__ #1
    • viccis #1
  5. 05

    Privacy is the real reason people pay

    The strongest defense of local inference was not cost or speed. It was control over sensitive personal and regulated data. People named journals, email, photos, medical records, and workplace constraints as reasons they will accept slower local models instead of cloud APIs. That makes local inference less of a price-performance hobby and more of a different product category with different buying criteria.

    If you are evaluating local AI for a team or product, treat privacy-sensitive workflows as their own market. The winning offer there is not cheapest tokens. It is acceptable performance with credible data isolation.

      Attribution:
    • wolvoleo #1
    • rbanffy #1
    • dannyw #1
    • Larrikin #1
  6. 06

    Old datacenter GPUs still punch above their price

    Several commenters pushed back on the idea that a pricey Mac is the obvious way to run 27B-class models locally. They cited MI50, MI25, dual 3090, and A6000-class setups delivering much higher throughput at lower cost, especially when tuned with tensor parallelism, RCCL, HIP, or Dflash2. The catch is obvious. You trade Apple's unified memory convenience for much more operational fiddling.

    If your target model fits in commodity or used GPU memory, build a price-performance comparison against refurb datacenter cards before buying a high-RAM Mac. Macs make more sense once capacity and convenience outrank raw throughput.

      Attribution:
    • wolvoleo #1
    • fotcorn #1
    • oceanplexian #1

Against the grain

  1. 01

    Maybe the hardware really is the wrong target

    A few people rejected the "just a software bug" framing and argued the bigger mistake is expecting strong local LLM performance from older Apple silicon in the first place. Their point was blunt: if you care about top-end throughput, buying into M3-era hardware and then complaining about not reaching 100 tok/s misses how fast the hardware curve is moving. In that view, the disappointing benchmark is a buyer expectation problem as much as a software problem.

    If throughput is the primary requirement, set a minimum performance target before choosing a platform. Do not back into hardware from brand preference and hope the software will save it.

      Attribution:
    • bilbo0s #1 #2
  2. 02

    Even the polished tools are not beginner simple

    The call for a Draw Things equivalent got pushback from someone who had used Draw Things and still found it confusing. The issue was not only app polish. It was the inherent cognitive load of model choice, quantization, prompt quality, and outdated recommendations. That is a useful warning against pretending local AI becomes mainstream once the UI improves.

    A better interface will help, but onboarding still has to narrow model choice and explain tradeoffs in plain language. If your product exposes every model knob upfront, you are still selling to enthusiasts.

      Attribution:
    • vunderba #1
    • e28eta #1
    • liuliu #1
  3. 03

    Sub-100 tok per second is not automatically useless

    The harsh claim that anything under 100 tok/s is worthless did not hold up. People running coding and batch-style workflows said 50 to 80 tok/s is already usable, and even lower speeds can be fine when privacy or asynchronous processing matters more than chat responsiveness. The better dividing line was around task type and context behavior, not a universal speed threshold.

    Match your latency target to the job. Interactive chat, coding agents, and overnight personal-data processing have different performance floors, so one token-speed rule will mislead you.

      Attribution:
    • ActorNightly #1
    • 0x457 #1
    • wolvoleo #1
    • zaj00l #1

In plain english

Dflash2 ↩
A speed optimization method referenced by commenters for boosting local model throughput on some GPU setups.
GGUF ↩
A file format commonly used to package quantized language models for llama.cpp and related tools.
HIP ↩
Heterogeneous-computing interface for portability, AMD's programming layer for GPU compute.
llama.cpp ↩
An open source C and C++ inference engine for running language models locally on CPUs and GPUs.
LLM ↩
Large Language Model, a machine learning model trained to generate and analyze human-like text.
MLX ↩
Apple's machine learning framework for running and training models on Apple silicon.
MTP ↩
Multi-token prediction, a speculative decoding method where a model predicts multiple future tokens to increase generation speed.
Ollama ↩
A popular local LLM runtime and packaging tool that exposes models through a simple interface and API.
oMLX ↩
A local model serving tool built around MLX that commenters referenced for faster inference on Macs.
prefill ↩
The stage where a model processes the input prompt and context before it starts generating output tokens.
quant ↩
Quantization, a method for shrinking model size and speeding inference by storing weights in lower precision.
RCCL ↩
Radeon Collective Communications Library, AMD's library for multi-GPU communication.
ROCm ↩
Radeon Open Compute, AMD's software platform for GPU computing and AI workloads.
tok/s ↩
Tokens per second, a common measure of how quickly a language model reads or generates pieces of text.
Vulkan ↩
A low-level graphics and compute API used for high-performance rendering.

Reference links

Privacy-focused model access

Model references

Cloud model speed comparisons