HN Debrief

Compression is prediction

  • AI
  • Machine Learning
  • Infrastructure
  • Programming

The post explains a standard information theory idea in plain language: good compression works by assigning short codes to likely next symbols, which means it needs a good predictor of what comes next. It walks through entropy coding, arithmetic coding, and then connects that logic to language models, where next-token prediction can also be seen as building a compressed representation of training data.

Treat compression-based framing as a sharp mental model for language models and probabilistic systems, not a blanket theory of intelligence or generalization. If you build models, evaluate them on distribution shift and task-specific edge cases instead of assuming better compression automatically means broader understanding.

Discussion mood

Mostly positive and engaged. People liked the post as a clear explanation of a classic idea, but many were irritated by how often this framing gets presented as fresh insight and insisted on more precision around novelty, attribution, and limits under distribution shift.

Key insights

  1. 01

    Distribution shift breaks the slogan

    The clean equivalence only holds when the future looks like the data your compressor modeled. A code that is optimal for one distribution can be badly wrong for another, even when both share the same alphabet and much of the same structure. That is the same failure mode as overfitting in machine learning, and it is exactly why next-token success does not automatically imply broad generalization.

    Use compression-style metrics as an in-distribution signal, not a substitute for out-of-distribution testing. If rare corners or adversarial cases matter, measure them directly.

      Attribution:
    • ssivark #1
    • gr_norm #1
    • omneity #1
  2. 02

    Compression can recover latent generators

    The strongest pro-LLM argument was not "models memorize well" but that enough compression pressure can back out a compact process that generated the data. The orbital mechanics example makes the point cleanly. A system that compresses many trajectories may converge on Newtonian structure, then extrapolate to new comets because it captured the generator rather than the samples. That framing better matches why language models can recombine concepts and why interpretability work finds abstract internal features instead of just phrase lookup.

    When judging whether a model can produce genuinely new outputs, ask whether the task has a compact underlying generator the model could infer. Expect stronger novelty where the data is governed by reusable structure, and weaker novelty where it is mostly contingent detail.

      Attribution:
    • jbay808 #1
    • SgtBastard #1
    • tptacek #1
  3. 03

    Classic compressors are already probabilistic models

    The article reads less surprising once you know how practical codecs work. Deflate, LZMA, Zstandard, JPEG, and similar systems are full of implicit or explicit bets about what comes next. LZ stages assume reuse of recent patterns, then entropy coders like Huffman, range coding, or Finite State Entropy turn those probability estimates into shorter bitstrings. Even examples that look like mere dictionary tricks still cash out as predictions about repetition or likely coefficient values.

    Do not treat compression theory and model theory as separate worlds. If you work on ML systems, old codec design is a useful source of intuition about probability modeling, representation, and what your objective is really rewarding.

      Attribution:
    • variadix #1
    • duskwuff #1
    • pornel #1
  4. 04

    Any compressor can act like a generator

    A useful operational point is that a compressor can be turned into an autoregressive model by scoring which candidate next token compresses best with the current context. That means the relationship is not just metaphorical. You really can generate text with gzip this way, it is just poor because gzip has a crude model. The gap between gzip and a transformer is model quality, not a different underlying principle.

    If you want to reason clearly about model behavior, separate the shared objective from the capacity of the model class. Better generation comes from a better probability model, not from escaping prediction.

      Attribution:
    • davmre #1
    • Legend2440 #1
    • throwaway_7274 #1
  5. 05

    The idea is old and well developed

    Several people grounded the post in a long line of prior work rather than treating it as an AI-era insight. Shannon established the bit-probability link, David MacKay built a whole course and book around the union of information theory and learning, and Jürgen Schmidhuber tied compression progress to curiosity and creativity years ago. The current wave feels new mostly because LLMs made these abstractions visible to a much wider audience.

    If this framing is useful to you, go to the source material instead of stopping at blog-post intuition. MacKay and Shannon give the durable foundations that current AI commentary keeps rediscovering.

      Attribution:
    • farfatched #1
    • melenaboija #1
    • rrherr #1

Against the grain

  1. 01

    The post overstates discovery vibes

    A skeptical line was not about the technical content but about presentation. The complaint was that polished explainer posts increasingly flatten established theory into a tone that sounds like fresh revelation, often without enough explicit lineage to prior teachers and sources. That leaves readers who do not know the field thinking they just witnessed a new deep insight instead of a well-known one.

    If you publish educational writing in mature fields, over-cite and name your intellectual ancestry early. It builds trust with experts and gives newcomers a path beyond the polished summary.

      Attribution:
    • chermi #1
    • mpalmer #1
    • exe34 #1
  2. 02

    Compression is closer to recall than foresight

    One dissenting view was that calling compression "prediction" sneaks in more agency than the examples deserve. On this account, lossy models mostly encode likely continuations from past data, which is strong probabilistic recall, not the kind of forecasting people mean when they talk about weather, markets, or genuine novelty. The distinction softens once prediction is defined statistically, but the objection is a good warning against anthropomorphic language.

    Be careful with product claims that turn next-token competence into broad forecasting ability. Reserve stronger language for systems that prove themselves on real unseen events, not just plausible continuations.

      Attribution:
    • caust1c #1 #2
  3. 03

    Compression alone can be philosophically empty

    A more abstract pushback said the slogan becomes vacuous if you ignore constraints on the compressor itself. An arbitrarily powerful search over possible generators will eventually find a tiny description for any computable sequence, so invoking "compression" by itself does not explain intelligence or world models. The important question is what model class, compute budget, and inductive biases make useful compression feasible in practice.

    When someone uses compression as an explanation, ask what resource limits and architecture make it nontrivial. Without those constraints, the claim sounds deeper than it is.

      Attribution:
    • sigbottle #1

In plain english

arithmetic coding ↩
A compression method that encodes a whole message as a range inside the number line based on symbol probabilities.
Deflate ↩
A common compression format used by gzip that combines Lempel-Ziv matching with Huffman coding.
distribution shift ↩
A situation where the data a system sees in use differs from the data it was trained or tuned on.
entropy coding ↩
A family of compression methods that assign shorter bit patterns to more likely symbols and longer ones to less likely symbols.
gzip ↩
A widely used file compression tool and format based on the Deflate algorithm.
Hutter Prize ↩
A long-running compression contest based on shrinking a fixed snapshot of Wikipedia, created around the idea that better compression reflects better intelligence.
in-distribution ↩
Data that comes from the same kind of source or pattern as the data used to train or model a system.
information theory ↩
The study of how information can be measured, encoded, transmitted, and compressed.
interpretability ↩
Research aimed at understanding what internal parts of a machine learning model represent and how they affect outputs.
JPEG ↩
A common image compression format that uses lossy steps to discard visual detail people are less likely to notice.
LLM ↩
Large language model, a machine learning model trained to predict the next token in text.
LZ ↩
Lempel-Ziv, a family of compression methods that shrink data by reusing repeated patterns.
LZMA ↩
Lempel-Ziv-Markov chain algorithm, a compression method used by xz that aims for high compression ratios.
next-token prediction ↩
The task of choosing the most likely next piece of text given the text that came before it.
Zstandard ↩
A modern compression algorithm, often shortened to zstd, designed for good speed and compression.

Reference links

Foundational texts and lectures

Popular explainers

Compression and AI research

Related concepts and side references