CoDistill-GRPO improves small models in GRPO by distilling knowledge from a larger model.
problem Small models in GRPO struggle with sparse rewards on difficult tasks.
method Simultaneously trains a large and small model using co-distillation and GRPO objectives.
result Significant improvement in small model performance over standard GRPO on mathematical benchmarks.