HN Debrief

They’re made out of weights

  • AI
  • Philosophy
  • Machine Learning
  • Culture

The post is a close adaptation of Terry Bisson’s 1991 short story “They’re Made Out of Meat,” swapping “meat” for neural network weights to make the familiar point that a strange substrate does not by itself rule out intelligence or consciousness. It leans on current AI language, saying there is no little homunculus, no explicit dictionary or grammar inside the model, just layers of learned numbers that somehow produce language. The author also disclosed that AI helped draft and proof the piece, which became part of the argument for many readers rather than a footnote.

If you work around AI, separate three questions that keep getting collapsed: what transformers are technically made of, what capabilities they already show, and what would count as evidence of consciousness. The practical risk is not just getting the philosophy wrong, but letting rhetoric and product incentives fill in gaps where definitions and tests are still missing.

Discussion mood

Mixed but engaged. Many readers enjoyed the piece as a clever homage and a useful way to explain how LLMs store learned behavior in parameters, but the dominant mood was skeptical of the jump from that observation to anything like consciousness or sentience. A second strong current was annoyance that the post leaned so heavily on a famous story and used AI assistance while gesturing at originality and depth.

Key insights

  1. 01

    Weights are not the whole system

    The useful correction is that a transformer is not literally “just weights.” The parameters matter most, but behavior also depends on tokenization, embeddings, attention, nonlinear activation functions, and the inference process that applies them. That does not kill the post’s main idea. It sharpens it. Grammar and semantics are not stored in a separate symbolic module, but neither are they reducible to a bare pile of numbers divorced from architecture and runtime.

    When you explain or evaluate an LLM, talk about the trained model plus the architecture and inference stack, not the weights in isolation. If your team says “it’s all in the weights,” make sure they are not quietly dropping the parts that actually make the computation work.

      Attribution:
    • phire #1
    • ozgung #1
    • yencabulator #1
    • throw310822 #1
    • FeepingCreature #1
    • mr_toad #1
  2. 02

    The story works as a bias check, not evidence

    The strongest defense of the piece was not that it proves LLMs are conscious. It is that it exposes how quickly people assume substrate decides the answer. That is a narrower and better claim. A model sounding person-like does raise the prior that something interesting is happening, and dismissing that with “it’s just statistics” is not an argument. But the story only earns a warning against overconfidence, not a conclusion about sentience.

    Use this framing to keep your own org honest. It is reasonable to reject strong claims of sentience while also rejecting glib dismissals that treat implementation details as a knockout argument.

      Attribution:
    • maxbond #1 #2 #3
    • erg0s4m #1
  3. 03

    Why the toaster analogy fails and still matters

    A long subthread clarified what is actually at stake in the “you could say this about a toaster” objection. The rebuttal is that LLMs are a live candidate in a way toasters are not, because they already produce open-ended language and social behavior that people naturally associate with minds. That makes the analogy more compelling than a random appliance story. The counter-rebuttal is equally important. Compelling rhetoric is not evidence. The adaptation borrows force from the original because humans already know meat hosts minds. That transfer does not automatically carry over to weights.

    Be suspicious of arguments that are persuasive because they preserve the emotional shape of a known truth. In product and policy conversations, ask what new evidence would survive if you stripped away the literary framing.

      Attribution:
    • maxbond #1
    • iainmerrick #1
    • Planktonne #1 #2
    • wredcoll #1
  4. 04

    Identity and memory are the weak points

    Several comments pushed past abstract philosophy and hit a concrete gap in current LLM systems. Human consciousness appears tied to continuity, locality, memory, and a stable embodied perspective. Deployed LLMs usually do not have that. Sessions reset, weights are static during inference, and the same model instance can serve unrelated users across machines and contexts. You can argue that the model specification is the identity, but that still leaves current systems looking more like repeated fragments than a persisting self.

    If you care about whether models might become moral patients, watch persistent memory, long-lived state, and self-updating behavior more than chat fluency. Those features are far more relevant to continuity-of-self arguments than another bump in benchmark scores.

      Attribution:
    • narrator #1
    • anon291 #1
    • overgard #1
    • ForceBru #1
  5. 05

    Geometry matters more than raw numbers

    One technical comment cut through the fixation on numbers by pointing out that trained models are really defined by geometry. You can rotate embeddings and internal matrices and preserve the same behavior. That means the interesting object is not a specific list of floating point values, but the relational structure they encode. This is a better mental model for why interpretability is hard and why “the model is just a big table of numbers” is true but uninformative.

    When you think about model portability, distillation, or interpretability, focus on preserved structure and function rather than literal parameter values. Two models can differ numerically and still implement the same internal organization for the tasks you care about.

      Attribution:
    • anon291 #1
  6. 06

    Pen-and-paper execution cuts both ways

    One skeptic argued that because a transformer’s next token can in principle be computed by hand from its weights, architecture, and prompt, it is hard to see where consciousness would enter. The stronger reply was not that this disproves machine consciousness, but that it forces a real commitment. If you believe consciousness is substrate-independent, then the same process carried out on a GPU, on paper, or in another medium should not automatically change the answer. That makes many pro-consciousness intuitions much harder to hold consistently.

    If your view depends on “the hardware somehow matters,” say how. If it depends on substrate independence, be ready for weird consequences that extend beyond GPUs to any faithful implementation of the same computation.

      Attribution:
    • Ajedi32 #1
    • gobdovan #1
  7. 07

    Lab incentives distort the consciousness debate

    A sharp practical point was that AI companies are financially exposed whichever way this conversation goes. If systems are framed as mere tools, companies avoid questions about rights and duties. If systems are framed as quasi-agents, they gain mystique and user attachment. That means both overclaiming and underclaiming can be strategic. The comments did not prove bad faith, but they did make clear that corporate messaging is not a neutral guide to what these systems are.

    Treat company narratives about model agency the way you treat benchmark marketing. Ask what legal, labor, and product incentives are being served before you treat the framing as a scientific position.

      Attribution:
    • gibspaulding #1
    • the_af #1
    • DangitBobby #1
  8. 08

    Derivative form and AI assistance changed the read

    For many readers, the most revealing line was the note that weights helped draft and proof the piece. That turned the adaptation into an example of the thing critics dislike most about AI output. It borrowed heavily from a classic work, then used the borrowed form to argue for the significance of the technology that helped produce it. A separate thread also noted the original author’s request not to adapt the story without checking first. Even readers who liked the homage said this muddied the post’s moral and artistic footing.

    If you publish AI-assisted work that leans on a recognizable source, disclose the assistance prominently and expect the disclosure to become part of the argument. In startups especially, provenance and permission can overwhelm the intended message.

      Attribution:
    • namuol #1 #2
    • f_klem #1
    • mikewarot #1

Against the grain

  1. 01

    Training data shape rule diffusion more than architecture

    One technically informed contrarian argued that the article overstates what is special about weights. On this view, rules only look smeared across a model because natural language data are messy and weakly structured. In cleaner domains, learned rules can become sharply interpretable, which makes neural networks look less mysterious and less like an alien substrate. The implication is that “they’re made out of weights” is not a deep ontological point so much as an artifact of current training regimes.

    Do not treat opacity as a permanent property of neural nets. In narrow domains with cleaner structure, interpretability may improve a lot, which should temper grand claims built on current fuzziness.

      Attribution:
    • benlivengood #1
    • noosphr #1 #2 #3
  2. 02

    The adaptation is mostly rhetorical freeloading

    A substantial minority thought the post was being mistaken for an insight when it was really a borrowed mood. Their point was not that substrate bias is impossible. It was that the post wins emotional credit from Bisson’s original without adding evidence or a new concept. The AI-assisted drafting note made this harsher. To them, the piece accidentally demonstrates the exact weakness critics complain about, derivative output that sounds profound while misunderstanding the source material’s force.

    If a claim feels powerful because it echoes a classic argument, ask what was actually added. In your own writing and product storytelling, novelty of framing is not the same as novelty of evidence.

      Attribution:
    • Planktonne #1
    • mjg2 #1
    • irishcoffee #1
    • overgard #1
  3. 03

    Simulating function may still miss the phenomenon

    One late comment pushed a harder philosophical objection with the kidney analogy. A perfect simulation of a kidney does not produce urine on your desk, so a perfect simulation of conversational behavior may still miss whatever consciousness is. This cuts against the dominant substrate-independence mood by insisting that matching the formal process is not enough if the target property depends on the physical realization itself.

    For high-stakes decisions, keep open the possibility that behavioral equivalence and phenomenological equivalence are different targets. That does not settle the issue, but it should keep you from treating imitation as proof.

      Attribution:
    • siavosh #1

In plain english

attention
A mechanism in transformer models that lets the system weigh which earlier parts of the input are most relevant when producing the next output.
GPU
Graphics Processing Unit, a processor that is often used for parallel math workloads like machine learning.
inference
Running a trained model to generate outputs from new inputs.
interpretability
The effort to understand what internal parts of a model represent and how they produce behavior.
LLM
Large language model, a machine learning system trained on large amounts of text that can generate and analyze language and code.
token
A chunk of text a model reads or generates, used for both pricing and context limits.
transformer
A neural network architecture used in modern language models that processes input tokens with attention mechanisms.

Reference links

Original work and adaptations

Technical explainers and papers

Interpretability and neuroscience critiques

Consciousness and philosophy references

Related books and media