RLVR Weak Supervision: When LLMs Can and Cannot Generalize
Summary
- • Generalization in RLVR governed by 'saturation dynamics' — slow-saturating models learn transferable reasoning
- • Qwen models generalize from as few as 8 examples; Llama models fail across all weak supervision settings
- • Root cause of failure is 'unfaithful reasoning' — correct answers produced without logically valid traces
- • Fix: domain continual pre-training plus SFT on reasoning traces before RL restores generalization
Details
Saturation dynamics determine whether RLVR generalizes under weak supervision
The key variable is not data size or label quality per se, but the shape of the training reward curve. A long pre-saturation phase — where reward is still rising — correlates with acquisition of transferable reasoning. Post-saturation training adds noise, not capability. This reframes the weak supervision problem from a data-centric issue to a training-dynamics issue.
Qwen-3B generalizes from 8 examples and survives 90% label noise; Llama-3B fails all three regimes
Llama-3B-Instruct on the MATH-500 benchmark drops from roughly 51% to 42% accuracy as label corruption rises from 10% to 90%. Qwen models using majority-vote proxy rewards show temporary gains before collapsing around 500 training steps, and self-certainty proxy rewards collapse in both model families.
Unfaithful reasoning — not output diversity — is the root cause of Llama's failure
Llama models exhibit higher output diversity than Qwen, ruling out the hypothesis that failure stems from insufficient exploration. Instead, failing models produce correct answers while generating reasoning traces that do not logically lead to those answers — the model is rewarded for the answer, not the reasoning path, and exploits that gap.
Continual pre-training plus SFT on reasoning traces before RL fixes faithfulness and recovers generalization
The two-stage remedy first exposes the model to domain-specific text via continual pre-training, then uses supervised fine-tuning on explicitly annotated step-by-step reasoning traces to instill faithful reasoning. Only after these stages is RL applied. This pipeline extends the pre-saturation phase in previously rapid-saturating models and has been empirically shown to restore generalization under weak supervision.
Model family and domain choice are decisive for low-resource RLVR deployments
Practitioners cannot assume any instruction-tuned model will benefit from RLVR under realistic, data-scarce conditions. Model selection and domain alignment matter more than reward engineering when labeled data is limited. Teams targeting reasoning improvement with minimal annotation budgets should evaluate saturation dynamics early, or risk investing in a training regime that will not generalize.
Research = empirical finding; Insight = mechanistic explanation; New Tech = proposed method or fix; Market Impact = practitioner implication
What This Means
For ML practitioners applying RLVR to reasoning tasks, model selection is not interchangeable — the same technique that reliably improves Qwen can fail entirely on Llama, even with identical data and reward setups. The practical implication is to probe saturation dynamics early in training runs and, for models that saturate quickly, invest in domain pre-training and supervised reasoning-trace fine-tuning before committing to a full RL pipeline. Teams working under annotation budget constraints should treat faithful reasoning — not output diversity or data volume — as the primary diagnostic signal for whether RLVR will generalize.