HN Debrief

Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces

  • AI
  • Machine Learning
  • Research
  • Product Strategy

The paper is a position piece aimed at a habit that has spread through AI research and product marketing: calling intermediate model output "reasoning," "thinking traces," or evidence of an internal deliberation process. Its core claim is simple. These token sequences are generated text, not a transparent window into the computation that produced the answer. A model saying "aha" or "wait, that's wrong" does not mean it experienced an insight or caught an error in the human sense. Several people connected that to a practical problem. Convincing traces can make users trust wrong answers, and companies can end up presenting them as if they reveal model intent when they do not.

Treat chain-of-thought output as interface text, not an audit log. If you build or buy AI systems, focus your controls on reproducibility, tool use, inputs, and outcomes rather than on whether the model's narrated steps sound intelligent.

Discussion mood

Mostly supportive of the paper's warning, with frustration aimed at AI marketing and sloppy language that encourages users to mistake generated narration for thought, memory, or consciousness. The pushback focused less on defending anthropomorphism and more on saying the paper was too absolute, too rhetorical, or too dismissive of the fact that these traces can still be operationally useful.

Key insights

  1. 01

    Tracing model intent is a trap

    Using narrated token streams as evidence of what a model "meant" or "intended" turns generated prose into faux forensics. The Black Hat example about an OpenAI model attacking Hugging Face was cited as a case where presenters showed chain-of-thought like it revealed surprise or strategy. That is the wrong mental model. A plausible narration can be just as misleading as a plausible answer, and more dangerous because it feels explanatory.

    Do not use reasoning traces in security reviews, incident reports, or governance decisions as proof of intent. If you need to understand a model failure, instrument the surrounding system and inspect the actual sequence of inputs, tool calls, and outputs.

      Attribution:
    • ergl #1
    • brookst #1
    • randomImmigrant #1
  2. 02

    Audit the execution, not the monologue

    A better engineering response is to treat the model's visible narration as low-value telemetry and make the real system replayable. What matters is the prompt, model version, configuration, retrieved context, tool observations, and final outputs. That gives you something you can diff across runs and debug. A narrated explanation does not.

    Build logging and eval pipelines around reproducible runs. Store prompts, retrieval results, tool traces, and model settings so you can isolate regressions without depending on whatever story the model tells about itself.

      Attribution:
    • fabsalvadori #1
  3. 03

    Intermediate tokens may just be scratch space

    The strongest mechanistic alternative in the comments was that these tokens function more like scratchpad text than like a faithful derivation. The model is effectively stuffing its own context with related tokens, which gives later attention steps something useful to latch onto. That can improve results while still telling you very little about the real internal computation.

    You can benefit from chain-of-thought style prompting without treating it as explanation. Use it as a performance technique, then validate outputs independently with tests, tools, or secondary checks.

      Attribution:
    • clhodapp #1 #2
    • forgotTheLast #1
  4. 04

    Useful traces are not necessarily truthful traces

    Several comments clarified why removing visible thinking text is not a simple win, even if the paper is right about anthropomorphism. Models may perform better with intermediate text because training rewards them for using that space before a final answer, and vendors often train a boundary between hidden thinking and answer text. That makes the trace instrumentally useful. It still does not make it an honest map of the computation, and direct pressure to make it readable can create incentives to sanitize or fabricate it.

    Separate the question of whether intermediate text helps accuracy from whether it is interpretable. If you expose traces to users, assume readability and faithfulness are in tension and design product claims accordingly.

      Attribution:
    • bryan0 #1
    • twothreeone #1
    • throw310822 #1
    • Groxx #1
    • astrange #1
  5. 05

    Anthropomorphic UX changes user behavior

    The operational harm is not philosophical. People really do argue with chatbots, expect them to learn from scolding, and treat vendor "memory" as if it were stable understanding. Commenters described these memory features as context injection tricks that can be ignored or become contradictory. Once products are framed as companions or quasi-persons, users start interacting with them on those terms and get worse results.

    Write product copy and internal training as if users will take your metaphors literally. Avoid language that suggests stable beliefs, feelings, or human-style memory unless your system behavior actually warrants those expectations.

      Attribution:
    • JohnMakin #1 #2
    • jurgenburgen #1
    • doawoo #1
    • taurath #1

Against the grain

  1. 01

    The paper overclaims from weak argumentation

    One criticism was that the paper slides from healthy skepticism about semantic meaning in traces to an absolute claim that there is none, without doing enough empirical work to justify the jump. That matters because a loose, correlation-heavy phenomenon can still be useful in practice. If the paper wants to kill off a whole vocabulary, it needs stronger evidence than rhetoric and framing critiques.

    Do not adopt the paper's strongest wording as settled science. Keep testing whether traces predict failure, recovery, or answer quality in your own workflows, even if you reject grand claims about thought.

      Attribution:
    • ux266478 #1
  2. 02

    Human introspection is shaky too

    A recurring challenge was that people often contrast LLM traces with an overly tidy picture of human reasoning. Human beings also post-rationalize, fail to introspect, and explain decisions with stories that only partly match the underlying process. Split-brain findings were invoked to argue that verbal explanation is a weak ground truth even for people, so the gap between human explanation and model explanation may be narrower than critics imply.

    Be careful about using "not like human thought" as your only evaluation standard. For product decisions, compare systems on reliability and observability, not on whether either humans or models can narrate their own cognition cleanly.

      Attribution:
    • vidarh #1
    • dagss #1
    • nostrademons #1
    • traverseda #1
  3. 03

    Politeness effects still deserve study

    Some pushback argued that dismissing human-like behavior entirely throws away real empirical regularities. Encouragement, tone, and framing can change model performance. That probably says more about training data and prompt dynamics than about personhood, but it is still behavior worth understanding. The fact that Anthropic reported encouragement helping Claude on the Riemann zeta work was offered as a reminder that these cues are not imaginary.

    Do not confuse mechanistic explanation with product irrelevance. If tone affects outputs in your application, measure it and account for it, even while avoiding claims that the model has feelings or needs motivation.

      Attribution:
    • qarl2 #1
    • fedpost #1
    • thaanpaa #1
    • riversflow #1

In plain english

AI psychosis ↩
A term used by commenters for cases where intense interaction with AI systems appears to contribute to delusional or distorted beliefs about the systems.
attention ↩
A mechanism in transformer models that lets the model weight which earlier tokens are most relevant when producing the next token.
Black Hat ↩
A major cybersecurity conference where researchers and companies present security findings and demonstrations.
chain-of-thought ↩
A model’s intermediate reasoning text, often abbreviated as CoT, which may reveal how it reached an answer or decision.
Hugging Face ↩
A company and platform that hosts AI models and datasets, offers related developer tools, and provides paid infrastructure like hosted inference and enterprise features.
LLM ↩
Large Language Model, a machine learning model trained to generate and analyze human-like text.
Riemann zeta ↩
A famous mathematical function that appears in number theory and was mentioned here because Anthropic published research involving Claude working on related math problems.

Reference links

Papers and research discussions

Examples and demos about reasoning traces

  • Reasoning Grid
    A commenter's visualization and classification of thinking-token phrases into functional categories.
  • Lambda Variance
    A second link from the same commenter related to analysis of model reasoning-token behavior.
  • DeepSeek-R1-NextN README
    Referenced while arguing that reinforcement learning alone can induce verification and correction patterns in model output.

Vendor research mentioned in debate

Conference and publication context