採用された応答と却下された応答に対する分類損失を用いたオフラインの選好最適化で、明示的な報酬モデルを学習させずに行う手法
Loading technique guide…
パターン: 2 段階学習: SFT → 明示的な報酬モデリングなしの直接選好最適化
理由: RLHF パイプラインを簡素化し、学習の安定性を高め、アライメント品質を保ちながら計算コストを削減
重要なポイント: 言語モデルを暗黙の報酬モデルとして扱い、分類損失で選好を直接最適化する
Simplified alignment without explicit reward modeling
Waiting for preference pairs...
Direct Optimization: Skip reward modeling and directly optimize the policy using preference data with a closed-form solution.
Bradley-Terry Model: Model human preferences probabilistically based on the difference in rewards between chosen and rejected responses.
KL Constraint: Maintain proximity to reference policy through KL divergence penalty (β parameter) to prevent overfitting.
Advantages: Simpler than RLHF, more stable training, no reward model needed, computationally efficient.
厳選された資料で理解を深めましょう
Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., 2023)
Training language models to follow instructions with human feedback (Ouyang et al., 2022)
Learning to summarize from human feedback (Stiennon et al., 2020)
Constitutional AI: Harmlessness from AI Feedback (Bai et al., 2022)