Comparison
Distillation vs Quantization
Distillation and Quantization are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for Distillation
Distillation comes up when the question is fundamentally about training.
DistilBERT: a 6-layer student of 12-layer BERT, 60% the size, 95%+ of the performance.
When you would reach for Quantization
When inference memory or speed is the binding constraint and you can tolerate a small quality drop.
Llama-3-70B-INT4 running on a 48 GB GPU instead of needing 2 A100s.
Frequently asked
What is the difference between Distillation and Quantization?
Distillation: Distillation trains a smaller "student" model to imitate the outputs of a larger "teacher" model. The student becomes much cheaper to run while retaining much of the teacher's quality. Quantization: Quantization reduces model weights from 16- or 32-bit floats to lower-precision types (INT8, INT4) so the model needs less memory and runs faster, usually with minor quality loss.
When should I use Distillation vs Quantization?
Distillation is the right concept when you are focused on training. When inference memory or speed is the binding constraint and you can tolerate a small quality drop.
Are Distillation and Quantization the same thing?
No. Distillation is training; Quantization is infrastructure. They are related but address different parts of the AI stack.