HN Debrief

Compression is prediction

  • AI
  • Machine Learning
  • Programming
  • Developer Tools

The post argues that compression and prediction are the same underlying game. A good compressor assigns short codes to likely next symbols, and a good language model assigns high probability to likely next tokens. The article walks through arithmetic coding, then connects that idea to large language models and to the broader claim that modeling the world well is what lets systems compress data well.

Treat compression metrics as a serious lens for evaluating model quality, especially for text, but do not confuse in-distribution compression with robust generalization. If you write technical explainers in fast-moving fields, add a short prior-art pointer so readers know whether they are learning a new result or a clean retelling of an old one.

Discussion mood

Positive on the article as an accessible explainer, but impatient with its framing. The mood mixed appreciation for the pedagogy with heavy eye-rolling from people who saw the idea as basic information theory being rediscovered under AI branding.

Key insights

  1. 01

    This idea predates LLMs by decades

    The compression-prediction link sits in classic information theory and algorithmic information theory, not in recent transformer work. The most useful pointers were David MacKay’s Information Theory, Inference, and Learning Algorithms, Solomonoff induction, Prediction by Partial Matching work, and the Hutter Prize, all of which make the same connection in more formal ways and show that people have treated compression as a proxy for intelligence for a long time.

    If this framing is useful to your team, anchor it in the older literature rather than AI-era blog discourse. That gives you better tools for thinking about limits, priors, and evaluation than treating it as a new insight from language models.

      Attribution:
    • farfatched #1
    • blahblahson #1
    • WithinReason #1
    • jll29 #1
    • ssivark #1
    • sethev #1
  2. 02

    Benchmark compression is not the same as generalization

    A compressor can win on a fixed corpus by exploiting quirks of that corpus without becoming a good model of future data. That is the same failure mode as overfitting in machine learning. The stronger framing is population versus sample. A model that compresses one dataset extremely well is not automatically the best predictor for held-out data unless you account for model cost, adaptation, and what distribution you actually care about.

    Use held-out compression as an evaluation tool, not raw benchmark compression alone. If you compare models this way, include model size and test on data that is plausibly out of training distribution.

      Attribution:
    • ssivark #1
    • gr_norm #1
    • versteegen #1
    • variadix #1
    • vrighter #1
  3. 03

    Compression explains why models can produce new outputs

    The best argument against the 'just next-token prediction' dismissal was that training is not mere memorization or local curve fitting. It is an inverse problem. The model is pushed to recover a compact process that could have generated the data. When that compact process tracks real structure, it can emit valid outputs that were not literally present in training, much like learning laws of motion from observations lets you predict a new comet.

    When evaluating model creativity or product usefulness, ask whether the system has learned a reusable generator of the domain rather than whether an exact answer existed in training. That is a more practical test for code, research, and design workflows than the stale memorization debate.

      Attribution:
    • throwaway_7274 #1
    • jbay808 #1 #2
    • adamgordonbell #1
  4. 04

    MacKay’s work gives the cleanest bridge

    Several people pointed to David MacKay as the best single source because his course and book tie together coding, inference, and learning without hype. Dasher was called out as a vivid example. It turns probabilistic text prediction into a user interface and is effectively a visual form of arithmetic coding. That makes the compression-prediction equivalence much easier to see than abstract AI arguments do.

    If you need to teach this idea internally, use MacKay’s materials or Dasher before jumping into LLM talk. They make the equivalence concrete with less philosophical baggage.

      Attribution:
    • smath #1
    • ballooney #1
    • jgraham #1
  5. 05

    The site itself undercut the message

    One high-signal side point was that the article page broke badly with JavaScript disabled, hiding text behind placeholder bars even though the content was in the source. That made the post feel needlessly heavy and ironic for a piece about compression. An ngrok engineer replied that it was a bug and would be fixed.

    For developer-facing content, basic HTML resilience still matters. If your audience includes technical readers, broken no-script rendering chips away at credibility faster than clever visuals add it.

      Attribution:
    • zahlman #1
    • dev_cprice #1

Against the grain

  1. 01

    Compression looks more like recall than prediction

    This view argues that calling compression 'prediction' smuggles in too much. A model can store statistical regularities and reproduce plausible continuations without actually forecasting novel events in the stronger sense people usually mean. Weather and markets were used as examples where past patterns alone do not buy reliable foresight, which makes the slogan feel overstated even if probabilistic recall is part of the story.

    Be careful when using this analogy in strategy or product messaging. It works best for modeling symbol sequences, not as a blanket claim that any strong compressor is a broadly predictive intelligence.

      Attribution:
    • kazinator #1
    • caust1c #1 #2
  2. 02

    Some compression uses full-sequence transforms

    Several commenters objected that not all compression is sequential next-step prediction. Many compressors transform the whole input, reorder it, or exploit global structure before entropy coding. That is still compression, but it is no longer natural to describe every step as predicting the next symbol. The clean equivalence holds most directly for statistical coders and autoregressive models, not for every trick in practical compression pipelines.

    When borrowing compression analogies for ML, be explicit about which class of compressors you mean. Statistical and entropy-coded schemes map cleanly to language modeling, while transform-heavy pipelines are a looser analogy.

      Attribution:
    • Lerc #1
    • ggm #1
    • tgv #1
  3. 03

    The zip-file metaphor hides model cost

    Some readers pushed back on treating LLMs as giant compressed archives because the analogy gets slippery once you account for parameter count and training data exposure. LLM-based compressors can beat traditional tools on English text, but that often excludes the cost of model weights. Static model files themselves are also still compressible, which underlines that 'the model is the archive' is not a precise accounting story.

    If you use compression to compare systems, define what counts in the bill. Include weights, dictionaries, and any side information, or the comparison will flatter large pretrained models in a misleading way.

      Attribution:
    • rrherr #1
    • cubefox #1
    • walrus01 #1

In plain english

arithmetic coding ↩
A compression method that represents an entire message as a range within a number line based on symbol probabilities.
entropy coding ↩
A family of compression methods that use shorter codes for more likely symbols and longer codes for less likely ones.
Hutter Prize ↩
A long-running competition focused on compressing a large Wikipedia dataset, often discussed as a proxy challenge for machine intelligence.
information theory ↩
The mathematical study of information, communication, compression, and uncertainty, founded by Claude Shannon.
inverse problem ↩
A task where you infer the hidden process or causes that produced observed data.
LLM ↩
Large language model, a machine learning system trained on huge amounts of text and code that can generate plausible responses and software.
PPM ↩
Prediction by Partial Matching, a statistical text compression technique that predicts the next symbol from recent context.
Shannon ↩
Claude Shannon, the mathematician and engineer who founded information theory.
Solomonoff induction ↩
A theoretical approach to prediction that favors simpler programs that could generate the observed data.
transformer ↩
A neural network architecture widely used in modern language models because it handles long-range relationships in sequences well.

Reference links

Foundational books and lectures

Compression and intelligence benchmarks

Neural compression and LLM papers

Historical and conceptual references

Popular explainers and essays