对被采纳和被拒绝的响应采用分类损失进行离线偏好优化，无需训练显式的奖励模型
Loading technique guide…
模式： 两阶段训练: SFT → 无需显式奖励建模的直接偏好优化
原因： 简化 RLHF 流程，提升训练稳定性，在保持对齐质量的同时降低计算开销
关键洞察： 将语言模型视为隐式奖励模型，通过分类损失直接优化偏好
Simplified alignment without explicit reward modeling
Waiting for preference pairs...
Direct Optimization: Skip reward modeling and directly optimize the policy using preference data with a closed-form solution.
Bradley-Terry Model: Model human preferences probabilistically based on the difference in rewards between chosen and rejected responses.
KL Constraint: Maintain proximity to reference policy through KL divergence penalty (β parameter) to prevent overfitting.
Advantages: Simpler than RLHF, more stable training, no reward model needed, computationally efficient.
通过这些精选资源加深理解
Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., 2023)
Training language models to follow instructions with human feedback (Ouyang et al., 2022)
Learning to summarize from human feedback (Stiennon et al., 2020)
Constitutional AI: Harmlessness from AI Feedback (Bai et al., 2022)