Comparison
Direct Preference Optimization vs Supervised Fine-Tuning
Direct Preference Optimization and Supervised Fine-Tuning are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for Direct Preference Optimization
When you have preference data and want a simpler pipeline than full RLHF.
Mistral-7B-Instruct-v0.2 was DPO-tuned.
When you would reach for Supervised Fine-Tuning
Supervised Fine-Tuning comes up when the question is fundamentally about training.
Training Llama-3-Base on Anthropic's HH-RLHF "chosen" responses as a first pass.
Frequently asked
What is the difference between Direct Preference Optimization and Supervised Fine-Tuning?
Direct Preference Optimization: DPO fine-tunes an LLM directly on (preferred, rejected) pairs without training a separate reward model or running RL. It is a simpler, more stable alternative to RLHF. Supervised Fine-Tuning: SFT is fine-tuning where each training example has an explicit input and a desired output, supervised by a loss that penalizes deviation from that output.
When should I use Direct Preference Optimization vs Supervised Fine-Tuning?
When you have preference data and want a simpler pipeline than full RLHF. Supervised Fine-Tuning applies when you are focused on training.
Are Direct Preference Optimization and Supervised Fine-Tuning the same thing?
No. Direct Preference Optimization is training; Supervised Fine-Tuning is training. They are related but address different parts of the AI stack.