一种无需参考模型的偏好对齐方法，在针对被选中回复的语言建模目标之外，为被拒绝的回复添加优势比惩罚项
Loading technique guide…
模式： 将有监督微调与基于优势比的偏好优化结合的单阶段训练
原因： 简化对齐流程、降低训练复杂度,并在无需独立奖励建模的情况下取得有竞争力的偏好对齐效果
关键洞察： 优势比惩罚项在有监督训练过程中直接优化偏好似然比,从而实现高效对齐
通过这些精选资源加深理解
ORPO: Monolithic Preference Optimization without Reference Model (Hong et al., 2024)
Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., 2023)
Training language models to follow instructions with human feedback (Ouyang et al., 2022)
Learning to summarize from human feedback (Stiennon et al., 2020)
Understanding the performance gap between online and offline alignment algorithms (Xiong et al., 2024)
A General Theoretical Paradigm to Understand Learning from Human Preferences (Xu et al., 2024)
The Alignment Problem from a Deep Learning Perspective (Christiano et al., 2023)
Reward Model Ensembles Help Mitigate Overoptimization (Coste et al., 2023)