人間の選好判断から学習した報酬モデルに対して、強化学習でポリシーをファインチューニングする手法
Loading technique guide…
パターン: 3段階の学習: SFT → 報酬モデル → 人間の嗜好に整合した PPO 最適化
理由: AI の振る舞いを人間の価値観に整合させ、安全性と有用性を高め、有害な出力を減らす
重要なポイント: 人間の嗜好 > 報酬モデル > 方策最適化 がスケーラブルな整合メカニズムを生む
Interactive RLHF training pipeline with human preferences
Waiting for generation...
Core Process: Generate multiple responses → Collect human preferences → Train reward model → Optimize policy with PPO while maintaining KL constraint.
Key Components: Supervised fine-tuning (SFT), reward modeling from preferences, proximal policy optimization (PPO) with KL penalty.
Human Feedback: Pairwise comparisons, rankings, or direct ratings that teach the model human values and preferences.
Benefits: Aligns model behavior with human values, improves helpfulness and safety, reduces harmful outputs.
厳選された資料で理解を深めましょう
RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (Bai et al., 2023)
Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., 2023)
Statistical Rejection Sampling Improves Preference Optimization (Liu et al., 2024)
Secrets of RLHF in Large Language Models (Zheng et al., 2024)