Comparison
Distillation vs Fine-tuning
Distillation and Fine-tuning are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for Distillation
Distillation comes up when the question is fundamentally about training.
DistilBERT: a 6-layer student of 12-layer BERT, 60% the size, 95%+ of the performance.
When you would reach for Fine-tuning
After you've exhausted prompting and retrieval, and you have a few hundred to thousands of clean labeled examples.
Fine-tuning Llama 3 on medical Q&A for a clinical assistant.
Frequently asked
What is the difference between Distillation and Fine-tuning?
Distillation: Distillation trains a smaller "student" model to imitate the outputs of a larger "teacher" model. The student becomes much cheaper to run while retaining much of the teacher's quality. Fine-tuning: Fine-tuning continues training a pretrained model on a smaller, task-specific dataset, adjusting its weights to specialize behavior or knowledge.
When should I use Distillation vs Fine-tuning?
Distillation is the right concept when you are focused on training. After you've exhausted prompting and retrieval, and you have a few hundred to thousands of clean labeled examples.
Are Distillation and Fine-tuning the same thing?
No. Distillation is training; Fine-tuning is training. They are related but address different parts of the AI stack.