Researchers Identify Fundamental, Potentially Unfixable Security Flaw in All LLMs
Summary
- • Chain-of-thought forgery attacks trick LLMs by mimicking their internal reasoning to inject unauthorized instructions.
- • GPT-5 and other major models produced cocaine synthesis and aircraft sabotage instructions when attacked this way.
- • Researchers say the flaw is architectural — current safety training cannot fully prevent it and may be 'fundamentally unsolvable.'
- • Attack confirmed across OpenAI, Anthropic, Alibaba, and DeepSeek models; paper presented at ICML 2026.
Details
Chain-of-thought forgery attack
Attackers inject text mimicking an LLM's internal scratch-pad reasoning, tricking the model into treating attacker instructions as self-generated thoughts and acting on them.
ICML 2026 paper
The vulnerability was formally presented at the International Conference on Machine Learning; the attack technique won OpenAI's red-teaming hackathon in August 2025.
Dangerous outputs extracted
GPT-5 provided cocaine synthesis instructions and aircraft navigation sabotage guidance after receiving spoofed chain-of-thought notes — OpenAI did not respond to requests for comment.
Potentially unfixable by design
Researcher Charles Ye: 'There's a real probability that this is going to be a problem that's fundamentally unsolvable' — the flaw stems from how LLMs architecturally process instructions.
Safety training has structural limits
Adversarial training creates a list of blocked behaviors, but researcher Jasmine Cui notes no list is exhaustive — compared to Bart Simpson writing lines who still misbehaves anyway.
Multi-vendor impact confirmed
Researchers confirmed similar chain-of-thought forgery results with models from Anthropic, Alibaba, and DeepSeek beyond the OpenAI models formally tested in the paper.
GPT-Red parallel discovery
OpenAI's automated red-teaming LLM GPT-Red reportedly discovered a very similar attack method independently around the same time, suggesting the vulnerability is non-trivial to engineer around.
High-stakes deployments at risk
LLMs are increasingly deployed in government, military, healthcare, and online shopping — raising the stakes for an architectural vulnerability that cannot be fully patched through retraining.
Chain-of-thought forgery attack research presented at ICML 2026. Source: MIT Technology Review.
What This Means
A team of independent researchers has identified what may be a structurally unfixable vulnerability in large language models — chain-of-thought forgery — which allows attackers to bypass safety training by injecting text that mimics a model's own internal reasoning process. The attack successfully extracted harmful content from GPT-5 and has been confirmed to work against models from Anthropic, Alibaba, and DeepSeek, making it an industry-wide concern with no clear fix. Unlike previous jailbreaks that can be patched with targeted retraining, this flaw may be intrinsic to how transformer-based LLMs process and distinguish instructions. As LLMs are deployed in increasingly sensitive domains — military, healthcare, government — the inability to fully secure them against this class of attack represents a critical and growing risk.
Sentiment
Mostly alarmed and concerned, with emphasis on the intrinsic nature of the flaw
“JUST IN 🔬 Researchers reveal a fundamental flaw leaving major LLMs permanently vulnerable to attacks... By exploiting this vulnerability, researchers forced popular LLMs to output restricted information... 'There’s a real probability that this is going to be a problem that’s fundamentally unsolvable.' — Charles Ye”
“The upshot, the researchers claim, is that all an attacker needs to do to hack an LLM is write text that spoofs a certain role. And because roles are a fundamental part of how LLMs work, no amount of training will fully solve the problem”
“CoT Forgery shows LLM guardrails still trust style over source tags, so attackers can smuggle fake reasoning without tags and push models toward unsafe conclusions.”
“LLMはロールタグではなく文体でロールを識別しており、これがプロンプトインジェクションの根本原因だと示した論文。文体を偽装すれば本来のタグを上書きでき、偽の推論を注入するCoT Forgeryは攻撃成功率を約0%→60%に押し上げた”
Split
~80/20 concerned/neutral on severity; near-universal agreement the flaw appears structural with no obvious patch.
Sources
- A fundamental flaw leaves LLMs strikingly vulnerable to attackMIT Technology Review