Researchers Identify "Matthew Effect" in RL Training for LLMs, Propose "Never Give Up" Fix
Summary
- • RL training boosts LLM performance on easy problems but provides little gain on hard ones
- • Researchers name this the 'Matthew Effect' — rich-get-richer compute allocation in RL training
- • Never Give Up (NGU) uses adaptive sampling to redirect compute toward harder problems
- • NGU improves per-compute performance on Deepscaler math and Manufactoria coding benchmarks
Details
Matthew Effect: RL over-invests compute in already-solvable problems
RL training shows large improvements on problems LLMs already handle well, but small improvements on genuinely hard problems. Authors name this the Matthew Effect — 'the rich get richer' — after the economic concept of cumulative advantage in network science.
Never Give Up (NGU): keep sampling until a correct answer is found
NGU is an adaptive sampling method that continues generating outputs for a given problem until one is correct. This naturally uses fewer samples on easy problems (solved quickly) and allocates more compute to harder ones — without any explicit difficulty scoring.
Asynchronous RL architecture enables dynamic per-problem compute allocation
NGU leverages asynchronous RL so problems are processed in parallel at variable rates. Easy problems finish quickly and free capacity; hard problems run longer. The paper discusses off-policy robustness as a key design consideration for this approach.
NGU improves per-compute performance on Deepscaler math benchmark
On Deepscaler, a math reasoning benchmark, NGU shows improved performance per unit of compute versus standard GRPO — with gains especially pronounced on harder problems in the benchmark distribution.
Manufactoria coding: NGU iteratively solves harder tests; GRPO fails outright
Standard GRPO with per-test rewards fails to fully solve Manufactoria problems that span a range of easy and difficult tests. NGU iteratively improves, solving increasingly difficult tests until it learns to fully solve the coding problems.
Based on arXiv preprint 'Learning to Solve Hard Problems in RL for LLMs by Never Giving Up' (Noukhovitch et al., submitted Sep 11, 2026).
What This Means
Standard RL post-training for LLMs has a hidden structural flaw: it preferentially reinforces things models already do well, providing diminishing returns on genuinely difficult problems. The "Never Give Up" method offers a simple asynchronous fix that dynamically redirects compute toward harder problems without added architectural complexity. Given that RL post-training is now central to frontier AI development, this finding could directly influence how labs design training pipelines — particularly as benchmarks saturate at easier difficulty levels and frontier progress increasingly depends on hard-problem performance.