Fine-Tuning Guide
Should You Fine-Tune At All?
Prompting, retrieval, distillation and fine-tuning fix different failures. Picking the wrong one is expensive, and the expense arrives weeks later as a model that behaves exactly as badly as before.
Start from the symptom
Most fine-tuning projects begin with a decision rather than a diagnosis. Match the complaint first:
symptom usually solved by
--------------------------------------- ---------------------------------
"it does not know our internal facts" retrieval, not training
"it gets facts wrong that change often" retrieval, not training
"it will not follow our output format" prompting, then fine-tuning
"it answers in the wrong voice" fine-tuning
"it cannot do our niche task at all" fine-tuning
"it is correct but too slow or costly" distillation to a smaller model
"it is right 90% of the time, not 99%" evaluation first: you do not know whyThe last row is the one that catches people. "Right most of the time" is not a symptom, it is a description of not having looked. Until you know which inputs fail and why, every remedy on this page is a guess with a bill attached.
What each option actually costs
prompting RAG fine-tuning distillation
change latency minutes hours days days
cost to iterate ~nothing low high high
fixes stale facts no yes no no
fixes format often no yes yes
fixes tone/voice partly no yes yes
cuts inference cost no no sometimes yes
needs labelled data no no yes yes (from teacher)Read the iteration row before the capability rows. Prompting can be changed and re-measured in minutes, so it should be exhausted first even when you are fairly sure it will not be enough. Fine-tuning turns every subsequent change into a training run, and that latency is what makes teams ship a model they already know is imperfect.
The three questions worth answering first
Do you have an eval? Not a vibe check: a fixed set of inputs with expected behaviour, that you can run before and after. Without it you cannot tell improvement from regression, and fine-tuning reliably produces both at once.
Do you have the data? The usual answer is "we have logs", and logs are not a dataset until they are deduplicated, decontaminated against the eval, quality-filtered and correctly templated. That work decides the result more than any hyperparameter, and it is where most of the outcome is won or lost.
Does the behaviour need to be stable? Anything that changes weekly (prices, staff, inventory, policy) belongs in retrieval. Training it in means retraining every time it moves, and being confidently wrong in between.
The honest ordering
For most teams, in most situations, the sequence that wastes the least money is: write the eval, fix the prompt, add retrieval if the failure is factual, and only then consider fine-tuning for behaviour that prompting could not reach. Distillation comes last and for a different reason: it is a cost and latency play once quality is already acceptable.
This ordering is boring and widely ignored, mostly because fine-tuning is the interesting one.
If the answer is yes, start with the data
The Dataset Lab covers the part that decides the outcome: exact and near-duplicate detection, decontamination against your eval set, quality filtering, truncation auditing, leakage-safe splitting and chat-template validation. Everything is implemented from scratch and runs on CPU, so you can see each mechanism work before pointing it at your own corpus.
For the mechanics once the data is ready, continue to Fine-Tuning Techniques, and read Risks and Open Questions before you ship anything you trained.