HF Daily Papers
· Papers
Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization
Reinforcement learning with verifiable rewards (RLVR) commonly uses entropy for advantage shaping. However, entropy cannot distinguish useful uncertainty from detrimental confusion, limiting its effectiveness as a correctness signal. We propose Contrastive Policy Optimization (CPO), which uses token