ラベル付きの入出力例からエージェントのコンポーネントを適応させ、明示的な検証とフォールバック動作を備える手法
Loading technique guide…
パターン: 対象ドメインのラベル付きデータで教師あり学習を行い、事前学習モデルを特定タスクに適応させる
理由: 既存のモデル知識を活かしつつ対象タスクに特化させ、妥当な学習コストで高い性能を実現する
重要なポイント: 事前学習モデルからの転移学習とタスク固有の教師信号の組み合わせが、汎化と特化の最適なバランスをもたらす
厳選された資料で理解を深めましょう
Fine-Tuning Pre-trained Language Models: Weight Initializations, Data Orders, and Early Stopping (Dodge et al., 2020)
How to Fine-Tune BERT for Text Classification? (Sun et al., 2019)
Universal Language Model Fine-tuning for Text Classification (Howard & Ruder, 2018)
Attention Is All You Need (Vaswani et al., 2017)
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., 2018)
Language Models are Few-Shot Learners (Brown et al., 2020)
T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., 2019)
RoBERTa: A Robustly Optimized BERT Pretraining Approach (Liu et al., 2019)
LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)
Prefix-Tuning: Optimizing Continuous Prompts for Generation (Li & Liang, 2021)
The Power of Scale for Parameter-Efficient Prompt Tuning (Lester et al., 2021)
AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (Zhang et al., 2023)
Domain-Adversarial Training of Neural Networks (Ganin et al., 2016)
Deep Domain Confusion: Maximizing for Domain Invariance (Tzeng et al., 2014)
Unsupervised Domain Adaptation by Backpropagation (Ganin & Lempitsky, 2015)
AdaBound: Adaptive Gradient Methods with Bound for Domain Adaptation (Luo et al., 2019)
QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023)
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (Zhang et al., 2023)
Instruction Tuning for Large Language Models: A Survey (Zhang et al., 2023)
DoRA: Weight-Decomposed Low-Rank Adaptation (Liu et al., 2024)
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (Wang et al., 2018)
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (Wang et al., 2019)
Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models (Srivastava et al., 2022)
HELM: Holistic Evaluation of Language Models (Liang et al., 2022)
BioBERT: A Pre-trained Biomedical Language Representation Model for Biomedical Text Mining (Lee et al., 2020)
FinBERT: Financial Sentiment Analysis (Araci, 2019)
LEGAL-BERT: The Muppets Straight out of Law School (Chalkidis et al., 2020)
CodeBERT: A Pre-Trained Model for Programming Languages (Feng et al., 2020)