Complementing reinforcement learning with SFT through logit averaging in the post training of LLMs
arXiv:2605.20555v1 Announce Type: new Abstract: We introduce a novel method that averages the logits of a frozen reference policy (e.g., SFT) and a trainable policy, and incorporate the method into Group Relative Policy Optimization (GRPO). In contrast to Reinforcement Learning with Verifiable Rewards (RLVR) methods, our proposal does not involve a Kullback Leibler (KL) regularization or critic; the trainable policy and the reference anchor are coupled through the logit averaging structure to leverage the reasoning expertise of the trainable policy while maintaining the formatting advantage of
The continuous evolution of LLM post-training methods, particularly RLHF alternatives, drives current research to enhance model performance and stability without traditional RL complexities.
Improving post-training efficiency and stability for large language models directly impacts the capabilities and deployment of advanced AI systems across various applications.
This novel logit averaging method offers a potential path to more effectively combine the reasoning of trainable policies with the formatting consistency of SFT models, potentially simplifying LLM optimization.
- · AI researchers
- · LLM developers
- · AI-powered applications
- · Traditional RLHF methods
- · High-compute LLM fine-tuning approaches
Refined LLM post-training techniques lead to more robust and performant AI models.
Easier and more stable LLM development could accelerate the deployment of complex AI agents and applications.
The simplified optimization process could lead to a broader adoption of custom LLMs in various industry sectors.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.LG