The paper demonstrates a simple but damaging trick against “encrypted reasoning” features in proprietary LLM APIs. Frontier models from OpenAI, Anthropic, and Google return opaque blobs that preserve hidden chain-of-thought across turns. The researchers replayed those blobs into weaker sibling models that could still read the underlying plaintext, then prompted the weaker models to transcribe it. That exposed the stronger model’s hidden reasoning without directly breaking the stronger model’s safeguards. The authors say providers patched the issue after disclosure.
The useful consensus was that the crypto itself was never the real boundary. The boundary was policy and product design. Providers wanted conversations to stay portable across models and sometimes across
stateless API calls, so some component had to turn those blobs back into readable context for another model. Once any weaker model can see the plaintext, a
jailbreak can exfiltrate it. That is why many people pushed back on proposed fixes like “just store it server side” or “just encrypt better.” If cross-model continuation remains, the plaintext still has to reach a model somewhere. The likely real fixes are uglier product choices like binding traces to one model, stripping reasoning on downgrade, or otherwise making model switching less seamless.
A second thread landed on what these traces are actually good for. People wanted them for prompt debugging, trust, contradiction checking, and understanding how an agent interpreted instructions. Others noted the traces are imperfect ground truth anyway. The recovered text may be a transcription by a weaker model, not a byte-for-byte dump, and vendor-visible summaries can already rewrite messy reasoning into cleaner narratives. That sloppiness also fed skepticism about benchmark claims. Several commenters read the paper’s examples as more evidence that public evals like
AIME get absorbed into training data, so polished “reasoning” on those tasks often reflects memorization or benchmark contamination as much as live problem solving.
The mood was also shaped by the politics around
distillation. A lot of commenters rejected the paper’s “stealing” frame outright and treated it as provider messaging designed to make model outputs feel like proprietary property even when users paid for the tokens and the outputs may not be copyrightable. Others were less interested in the legal wording than in the strategic pattern. Labs scraped the public web, now want strict controls on using their outputs, and hide internals that would help users audit or improve prompts. The sharper take was that this vulnerability matters less as a one-off jailbreak and more as a reminder that closed
API features are being designed around competitive control as much as safety or user value.
stolen-thoughts.com