Prompt first
Use prompting or few-shot examples when behavior can be expressed clearly and the added context still fits latency and cost constraints.
Fine-Tuning Guide
Fine-Tuning Cheatsheet
A framework-neutral checklist for controlled experiments. Hyperparameters and memory needs depend on the exact model, data, objective, and hardware; copy values only from a maintained recipe for that combination, then validate them in a pilot.
Retrieve or call a tool
Use retrieval for changing, private, or attributable knowledge and tools for calculations, actions, and authoritative system state.
Fine-tune deliberately
Fine-tune a repeated behavior only when you have representative examples, a sealed test set, and a measured baseline to beat.
Data contract
Before training
- • Define allowed sources, licenses, consent, retention, and deletion
- • Match real input distribution, language, format, and edge cases
- • Remove secrets, unnecessary personal data, duplicates, and corrupt examples
- • Review label policy and disagreements instead of hiding them
- • Deduplicate across train, validation, and test sets
Keep the test set sealed
- • Success, failure, boundary, safety, and out-of-distribution cases
- • Slice labels for important users, languages, and content types
- • A written scoring rubric and adjudication process
- • No examples generated from or tuned against test outputs
- • Versioned revisions so comparisons remain meaningful
Synthetic data rules
- • Prefer real production examples; generate data to fill gaps, not to replace collection
- • Seed generation with vetted real examples so outputs track your task, not the generator’s defaults
- • Filter generated examples at least as strictly as scraped data; the filter decides the value
- • Generate the rare and hard cases the real corpus lacks, then label them under the same policy
- • A few hundred to a few thousand clean examples beat a much larger noisy set
What to tune and watch
| Control | Why it matters | Signal to inspect |
|---|---|---|
| Learning rate | Sets update magnitude | Instability, slow learning, validation quality |
| Training duration | Controls exposure to examples | Held-out plateau and regression growth |
| Effective batch | Changes gradient noise and throughput | Steps per second, memory, convergence |
| Sequence length | Strongly affects memory and training distribution | Truncation, padding waste, long-input quality |
| LoRA rank and targets | Sets adapter capacity and placement | Trainable parameters, task quality, overfitting |
| Precision / quantization | Trades memory, compatibility, and numerical behavior | NaNs, speed, memory, exported-model quality |
Experiment record
- Base checkpoint
- Repository, exact revision, tokenizer, chat template, and license review
- Data
- Immutable manifest, provenance, filtering, deduplication, splits, and rights
- Objective
- SFT, preference optimization, or reinforcement objective and loss masking
- Update method
- Full tuning, LoRA, or QLoRA with target modules and adapter settings
- Optimization
- Learning rate, schedule, effective batch, steps, precision, and seed
- Environment
- Code revision, package lock, container, driver, accelerator, and topology
- Evaluation
- Holdout revision, rubric, slice metrics, safety suite, and base regression suite
- Artifacts
- Checkpoint hashes, adapter/merge method, serving template, and rollback target
Release gate
- The tuned candidate beats every required baseline on the sealed holdout.
- Per-slice results meet written thresholds and include sample counts.
- Base-capability, safety, privacy, and misuse tests stay within the regression budget.
- Serving uses the same tokenizer and template tested offline.
- Latency, throughput, memory, and total cost are measured on production-like infrastructure.
- A canary, monitoring owner, alert threshold, and rollback artifact are ready.
Stop the run when
Loss is non-finite or gradients remain unstable.
Validation quality degrades while training loss improves.
The data or license review uncovers an unresolved issue.
A checkpoint cannot be loaded, evaluated, or resumed reproducibly.
Safety or privacy checks cross the agreed threshold.
The pilot cannot plausibly meet the deployment cost or latency target.