HN Debrief

Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)

  • AI
  • Machine Learning
  • Research
  • Developer Tools

The paper studies chain-of-thought faithfulness in large language models, meaning whether the readable step-by-step explanation matches the process that actually produced the answer. Using paired questions and answer formats, it finds models often latch onto superficial template cues, decide on an answer early, then generate a convincing rationale for that answer even when the rationale contradicts the choice at the end. Several people said this matches day-to-day experience with models like Qwen, where the visible reasoning can look right until the last line flips to the wrong yes or no, or look wrong the whole way and still land on the correct answer.

Treat exposed reasoning traces as a noisy interface, not as ground truth about model internals. If you rely on them for evaluation, safety, or product UX, add tests for answer consistency under paraphrase and for whether traces predict behavior at all.

Discussion mood

Mostly unsurprised and skeptical of anthropomorphic language. The mood was that the paper confirms what practitioners already see in production, while sharpening an important point: readable reasoning traces are useful outputs, but they are not trustworthy evidence of how the model actually arrived at an answer.

Key insights

  1. 01

    Extra tokens help without faithful explanations

    Adding reasoning tokens seems to improve performance even when their literal content is partly broken or generic. That points to a mechanism where the model benefits from more computation steps or more passes over earlier context, not from the human-readable semantics of the trace itself. This is a stronger claim than “the traces can be misleading.” It says the visible explanation may be riding on top of a useful internal process rather than exposing it.

    Do not assume that improving or policing the wording of chain-of-thought will improve the underlying reasoning. Benchmark whether longer inference budgets help even when you hide, compress, or replace the visible trace.

      Attribution:
    • orbital-decay #1
    • 8note #1
    • rcxdude #1
  2. 02

    Semantic invariance is the better test

    A more revealing way to probe model reasoning is to ask whether answers stay consistent across different representations of the same underlying relation. If the model really tracks the structure of the problem, paraphrases and equivalent formulations should transform the answer and explanation in predictable ways. This shifts evaluation away from reading the prose of the trace and toward checking behavioral stability under reformulation.

    Add paraphrase and representation-shift tests to evals. If a model changes its answer when only the surface form changes, the explanation quality is a distraction.

      Attribution:
    • sergio_valencia #1
  3. 03

    Related work weakens the case for token-level semantics

    The linked paper "Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!" pushes the critique further by claiming correctness does not correlate strongly with valid traces, and that models trained on corrupted traces can still perform well. That matters because it attacks a common assumption behind chain-of-thought analysis, namely that better-looking intermediate text reflects better reasoning. If that assumption fails, using visible traces as a scientific probe becomes much shakier.

    When you evaluate reasoning models, separate answer quality from trace quality. A polished trace can be mostly theater, so trace-based reward models and audits need independent validation.

      Attribution:
    • stymaar #1

Against the grain

  1. 01

    The paper is chasing a bad metaphor

    Calling these outputs “thought,” “reasoning,” and “faithfulness” invites a category error. On this view, the surprising part is not that visible chain-of-thought fails to mirror internal computation. The surprising part is that anyone expected ordinary human notions of reasoning to transfer cleanly to token generation in the first place. That reframes the paper as cleanup for language the field should have been more careful with from the start.

    Be precise in product and research language. If your team talks as if model text is direct evidence of cognition, you will design the wrong evals and overclaim interpretability.

      Attribution:
    • Planktonne #1
    • swatcoder #1 #2
    • kibwen #1
  2. 02

    Human-style rationalization is still a useful analogy

    The pushback to anti-anthropomorphism was that post-hoc justification is not unique to models, and the analogy can still help explain observed behavior. The point is not that humans and LLMs work the same way. It is that rationalization after a decision is a familiar failure mode, so the model behavior should not be treated as bizarre or inexplicable. That analogy can guide better expectations even if it does not validate strong claims about machine thought.

    Use human analogies only as loose intuition for failure modes, not as evidence about mechanism. They can help teams anticipate errors, but they should not substitute for direct testing.

      Attribution:
    • Georgelemental #1
    • freejazz #1
    • cyanydeez #1

In plain english

chain-of-thought ↩
A model’s intermediate reasoning text, often abbreviated as CoT, which may reveal how it reached an answer or decision.
faithfulness ↩
In this context, whether a model’s visible explanation accurately reflects the process that actually produced its answer.
Qwen ↩
A family of open-weight large language models developed by Alibaba.

Reference links

Related papers on reasoning traces

Mechanistic interpretability references

Related discussions