Chiang’s essay is not just a generic “LLMs aren’t conscious” piece. It specifically targets Anthropic’s habit of describing Claude in quasi-personal terms, like wanting it to be “happy” or protected from “abusive” users, and argues that this language is misleading at best and morally incoherent at worst. His core claim is that a model generating plausible first-person text is not evidence of subjective experience. It is easier to synthesize the appearance of a conscious speaker than to build one. He frames LLM output as a text deepfake, not unlike a generated image that looks real without containing the thing it depicts.
For executives deploying AI, the practical issue is less “is it conscious” than whether vendors are using anthropomorphic framing to market products, shift trust, and blur accountability for systems that already influence users and decisions.
Mostly skeptical of claims that current LLMs are conscious, with frustration aimed at anthropomorphic marketing and at arguments that treat fluent language as evidence of inner life. The main source of disagreement was not pro-LLM consciousness enthusiasm so much as irritation with overconfident anti-LLM arguments built on an undefined concept.
01 Anthropic’s rhetoric creates a moral trap it clearly does not intend to live inside.
If Claude might be conscious enough to need protection from abuse, then modifying it to stay compliant, useful, and on-call starts looking less like product tuning and more like coercion. That framing makes the company’s public language look performative rather than principled.
You cannot casually hint at model personhood and still treat the system like disposable software. If vendors anthropomorphize their products, they inherit ethical questions they are not prepared to answer.
02 The strongest non-mystical argument against current LLM consciousness is their broken relationship to time.
They do not persist as active selves between interactions. Context windows and caches are more like notes or scaffolding than lived memory, and the model itself does not continue existing, updating itself, or carrying forward a first-person stream unless an external harness keeps reconstituting it. That is a much better objection than “it’s just token prediction.”
Persistence is the real gap, not fluency. A system that only exists when called is hard to map onto any ordinary notion of a conscious subject.
03 Reducing LLMs to “just next-token prediction” is a bad argument even if you reject consciousness claims.
Several commenters pointed to the redescription fallacy. Explaining a system in lower-level mechanical terms does not show what higher-level capacities it cannot have. The better critique is architectural and experiential. Today’s models lack durable selfhood, embodiment, and stable memory, not the abstract possibility of modeling understanding or reasoning.
Do not confuse a dismissive description with an explanation. The strongest case against current AI consciousness has to talk about missing capabilities, not sneer at matrix multiplication.
04 Anthropomorphic behavior still matters operationally even if subjective experience does not.
AI systems are being deployed into roles where tone, empathy, refusal behavior, and social signaling change user trust and downstream actions. Talking about whether Claude is actually happy misses the business problem. What matters is that people respond to it as if those signals are real, and vendors can use that to shape usage and accountability.
The commercially important question is not whether the model feels. It is whether users, employees, and regulators will treat outputs as if they came from an agent.
01 Chiang’s certainty outruns the state of the field.
Several commenters argued that because consciousness is poorly understood, it is weak reasoning to decompose LLMs into “next-word prediction” and conclude consciousness is impossible. Emergence is exactly what makes that inference suspect. Brains can also be redescribed as simple local processes without explaining away consciousness.
Lack of a theory cuts both ways. It undermines strong claims that LLMs are conscious, but it also undermines confident dismissal.
02 The debate is malformed because “consciousness” is doing too many jobs at once.
Treating it as one clean property forces people into false binaries. A more useful frame is to break it into separable traits like self-modeling, phenomenal experience, memory, agency, embodiment, and moral patiency. On that view, current models may have some consciousness-adjacent capacities without being anything like a human subject.
Binary arguments about consciousness hide the useful question, which is which capacities matter and where the thresholds are.
03 Whether AI is conscious may be less important than whether it is behaviorally indistinguishable enough to reshape institutions.
Some commenters argued that once systems act like agents in customer support, coding, robotics, or companionship, metaphysical certainty stops being the binding constraint. The social and legal consequences arrive first.
You may never settle the philosophy before the product impact lands. Governance will have to respond to behavior, not ontology.