What gets quantized
Weights first; activations and the KV cache can follow. Each additional target saves more memory and risks more quality, so they are separate decisions.
AI Inference Guide
Model Preparation
Work done to the model before it serves traffic. Three levers change what a deployment costs and how it behaves: shrink the numbers (quantization), shrink the model (distillation), or change the behavior (fine-tuning).
Quantization: lower precision, smaller everything
Quantization stores numbers in fewer bits. Since a parameter costs 4 bytes at FP32, 2 at FP16/BF16, 1 at FP8/INT8, and half a byte at INT4, a 7B model drops from about 28 GB to 14, 7, and 3.5 GB as precision falls. The win is threefold: less memory to buy, less data to move per token (decode is bandwidth-bound), and faster arithmetic on hardware with native low-precision support.
Named methods
Post-training approaches dominate serving: GPTQ (layer-wise error minimization), AWQ (protects the small fraction of weights that matter most), SmoothQuant (shifts difficulty from activations to weights for W8A8). Quantization-aware training exists but is rarer for LLMs.
In practice
Most teams download pre-quantized checkpoints (hub releases, GGUF Q4/Q5/Q8 levels for local runtimes) rather than quantizing themselves. Check that your GPU generation natively supports the target precision, or the speed benefit evaporates.
The cost is accuracy: usually small at 8-bit, noticeable sooner on smaller models and harder tasks. Never accept a quantized build on perplexity alone; run your own evaluation set against the full-precision baseline.
Distillation: train a smaller model on a bigger one
Distillation trains a compact student model to imitate a larger teacher, transferring capability into a cheaper artifact. Unlike quantization, which compresses the same model, distillation produces a genuinely different, smaller model with lower latency, memory, and per-request cost.
- Signal: the student can learn from the teacher's full output distribution (soft labels) rather than just final answers, and training typically blends imitation loss with standard cross-entropy.
- Variants: match final responses, match intermediate representations, or train on the teacher's reasoning traces so the student inherits step-by-step behavior.
- Where it shines: a well-scoped task where a large model is overkill per request; distilled small models are also how reasoning ability reaches deployable sizes.
- Caveats: it is a training project with real compute cost, the student will not match the teacher everywhere, and teacher licence terms may restrict training on its outputs.
Fine-tuning: change what the model does
Fine-tuning continues training an existing model on your own examples: domain language, output formats, tone, or task behavior that prompting cannot reliably produce. Reach for it after prompting, few-shot examples, and retrieval have been tried, and build the evaluation set before training so you can prove the tune helped.
Methods
- Full fine-tuning updates every weight: strongest shift, highest VRAM and cost
- LoRA trains small adapter matrices on frozen weights; QLoRA does so on a quantized base, reaching consumer GPUs
- Adapters can be merged into the weights or loaded per request at serving time, so one base model can host many behaviors
- Data beats volume: a few thousand clean, well-labeled examples outperform noisy bulk
Tooling
- Axolotl: config-driven training recipes
- Unsloth: memory- and speed-optimized kernels for single-GPU tuning
- Torchtune: PyTorch-native fine-tuning library
- LLaMA Factory: broad model support with a UI
- Hosted fine-tuning trades convenience for lock-in to the provider's lifecycle
This site has a dedicated hub covering techniques, frameworks, and local versus cloud training paths in depth: see fine-tuning.
Which lever, when
| Symptom | Reach for |
|---|---|
| Model does not fit the GPU, or serving cost is too high at acceptable quality | Quantization first; it is cheap to try and reversible |
| A narrow task where the large model is overkill per request | Distillation to a small student, or an existing small model plus fine-tuning |
| Wrong domain behavior, format, or tone despite good prompts and retrieval | Fine-tuning (LoRA first, full tune if adapters fall short) |
| All of the above at scale | They compose: a distilled or fine-tuned model is routinely quantized for serving |