NVIDIA / CUDA
Check the framework's CUDA, driver, compute-capability, precision, and optional-kernel requirements before installing packages.
PyTorch install selectorFine-Tuning Guide
Local Fine-Tuning Setup
Validate the exact model and training recipe on your hardware. Generic parameter-count-to-VRAM tables are unreliable because sequence length, precision, optimizer state, activations, adapters, attention kernels, and batching all change the result.
Choose a supported stack
AMD / ROCm
Verify the operating system, GPU, ROCm release, and framework support matrix; do not assume CUDA-only quantizers or kernels have an equivalent path.
ROCm installation docsApple silicon
MLX-based workflows use unified memory and a different toolchain. Confirm the exact model architecture and export path before committing to it.
MLX LM repositoryPreflight before a long run
- 1.Confirm driver, runtime, compiler, framework, and accelerator are mutually supported.
- 2.Download only the intended checkpoint revision and verify available storage.
- 3.Inspect tokenization and loss masking on a few formatted examples.
- 4.Run one forward/backward step and record peak device and host memory.
- 5.Overfit a tiny sample to catch broken labels, templates, and optimizer configuration.
- 6.Save, resume, load, and generate from a checkpoint before scaling the dataset.
- 7.Measure thermals and sustained throughput, not only the first few steps.
- 8.Keep enough disk and time headroom for checkpoints, logs, evaluation, and retries.
Memory and throughput levers
Reduce memory
- • Use a smaller checkpoint or adapter method
- • Reduce sequence length or micro-batch size
- • Enable supported mixed precision and checkpointing
- • Consider optimizer or state offload after measuring the penalty
Improve utilization
- • Profile data loading and sample-length distribution
- • Use packing only with correct masks and supported attention
- • Increase batch or accumulation only after correctness checks
- • Benchmark optional kernels on the actual GPU
Keep it reproducible
- • Pin package and model revisions
- • Capture the full configuration and random seeds
- • Version data manifests rather than copying ad hoc folders
- • Record hardware, driver, and environment details
Local does not mean automatically private
- Review model download terms and dataset rights before use.
- Restrict access to raw data, caches, logs, checkpoints, and exported adapters.
- Scan training data and artifacts for secrets and personal data.
- Encrypt backups and define deletion and retention procedures.