RLVR Narrows Solution Space at Entrance, Parameter Interpolation Restores Coverage
ORIGINAL / Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space
This study localizes the contraction of policy solution space in RLVR to the entrance stage, and shows that targeted interventions can recover diversity without sacrificing pass@1.
01 ABSTRACT
Using Countdown task with enumerable solution families, the authors show that RLVR reduces solution coverage by up to 67%, concentrated at entrance. Supplying entrance prefixes or interpolating late-layer parameters restores coverage, while entropy collapse is not inevitable.
02 KEY FINDINGS
- RLVR reduces solution coverage by up to 67%, halving even on problems solved across all checkpoints.
- Contraction concentrates at entrance; token likelihood shifts are 11-16x larger before first arithmetic operation.
- Providing unselected entrance prefix raises completion rates in low-access families from 0.018 to 0.212 (PPO).
- Late-layer parameter interpolation with early checkpoints increases coverage by 37% with no pass@1 loss.
- Early entropy collapse recurs across six math benchmarks but SFT baseline and staged pipelines retain entropy.
Return to the primary source
Read original ↗AI GENERATED SUMMARY / DISCOVERED BY ARXIV CS.CL · ARXIV CS.LG