AI Reasoning Models Achieve IMO Gold Medals but Researchers Find Evidence of Shortcut-Taking Over Genuine Logic
Summary
- • Large reasoning models (LRMs) have delivered stunning results — including IMO gold medals and solving open math research problems — but multiple studies find they may rely on surface-level shortcuts rather than generalizable reasoning.
- • Apple researchers labeled LRM chain-of-thought an 'Illusion of Thinking,' finding complete accuracy collapse under surprisingly simple conditions; Santa Fe Institute confirmed shortcut use on visual analogy puzzles.
- • Despite the critiques, Google DeepMind and mathematician Terence Tao used AI to rediscover or improve solutions to 67 math problems spanning geometry, combinatorics, and number theory.
- • Expert Melanie Mitchell (Santa Fe Institute) summarizes: LRMs improve accuracy in practice, but their chain-of-thought outputs don't faithfully reflect internal computation, and much of the generated text provides no useful signal.
Details
IMO gold medals & open problems solved
LRMs achieved IMO gold medal-level performance and, in May 2026, an OpenAI model solved a famous open mathematical research problem in a single attempt.
Apple 'Illusion of Thinking' critique
Apple researchers found LRM chain-of-thought is subject to complete accuracy collapse under surprisingly simple conditions, calling it an illusion rather than genuine reasoning.
Santa Fe Institute: surface-level shortcuts
Melanie Mitchell's team showed LRMs can crush carefully designed visual reasoning benchmarks using surface-level pattern-matching shortcuts rather than generalizable logic.
DeepMind + Tao: 67 math problems improved
Google DeepMind and mathematician Terence Tao used AI to rediscover or improve solutions to 67 problems across mathematical analysis, combinatorics, geometry, and number theory.
Chain-of-thought is not faithful
Mitchell's key finding: LRM chain-of-thought text doesn't reliably reflect what's happening inside the model, and much of it is not even useful for improving the answer.
'Jagged intelligence' framing
Researchers describe LRM capability as 'jagged' — extraordinarily strong on some tasks, inexplicably weak on others — with no reliable way to predict which is which in advance.
LRM reasoning debate — key research findings for and against genuine AI reasoning capability, mid-2026.
What This Means
The debate over whether AI reasoning models truly "reason" reflects genuine scientific uncertainty about what is happening inside these systems. LRMs produce impressive results on hard benchmarks yet fail simple tasks in ways inconsistent with true generalization — and their visible chain-of-thought steps appear to be post-hoc outputs rather than windows into actual computation. For practitioners, this creates a difficult trust problem: the models are powerful, but neither their outputs nor their reasoning traces can be fully relied upon. How this scientific debate resolves will have major implications for deploying AI in any domain requiring verifiable, auditable logic.
Sources
- Is AI reasoning right for the wrong reasons?Quantamagazine