Supervised fine-tuning (SFT)
Data: Input and desired-output examples
Teach a stable task, response structure, style, or tool-calling pattern.
Watch: The model may imitate errors and biases in demonstrations; unrelated behavior can regress.
Fine-Tuning Guide
Fine-Tuning Techniques
Make two separate choices: what learning objective matches your data, and how much of the model to update. No method wins across every model, task, and hardware stack.
1. Choose the learning objective
Preference optimization
Data: Preferred and non-preferred responses, or equivalent feedback
Shift choices where quality is easier to compare than to specify as one ideal answer.
Watch: Label policy, annotator agreement, and held-out evaluation matter as much as the algorithm.
Reinforcement fine-tuning
Data: Prompts plus a reliable grader or reward signal
Optimize outcomes that can be scored consistently across generated trajectories.
Watch: Reward hacking, grader bias, instability, and higher operational complexity require extra controls.
2. Choose the parameter-update method
Full-parameter tuning
Updates the base weights and offers the most flexibility, with the largest memory, storage, and rollback burden.
Consider only when adapter methods miss a measured target and the infrastructure and data justify it.
LoRA / adapters
Freezes the base model and trains smaller update modules, often under one percent of the parameter count. Artifacts are compact and multiple adapters can share one base.
A sensible first experiment for many transformer workloads; rank and target modules still need empirical tuning.
QLoRA
Backpropagates through a quantized frozen base into LoRA adapters, reducing base-weight memory at potential compatibility and throughput costs.
Useful when memory is the binding constraint; validate quantization support and final serving quality.
Variants are hypotheses, not upgrades
Adapter variants
DoRA, AdaLoRA, LoRA+, and other variants change parameterization, allocation, or optimization. Try one only when its implementation supports your model and it addresses a measured limitation of the simpler baseline.
Quantization-aware training
QAT targets quality after low-precision deployment. It is different from QLoRA, which primarily uses a quantized frozen base to reduce training memory. Validate the exported format on the intended runtime.