Comparison
Constitutional AI vs Direct Preference Optimization
Constitutional AI and Direct Preference Optimization are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for Constitutional AI
Constitutional AI comes up when the question is fundamentally about safety & alignment.
A constitutional principle: "Choose the response that is least harmful and most helpful."
When you would reach for Direct Preference Optimization
When you have preference data and want a simpler pipeline than full RLHF.
Mistral-7B-Instruct-v0.2 was DPO-tuned.
Frequently asked
What is the difference between Constitutional AI and Direct Preference Optimization?
Constitutional AI: Constitutional AI is Anthropic's alignment technique that uses a written set of principles ("constitution") plus AI feedback to shape model behavior instead of relying entirely on human labels. Direct Preference Optimization: DPO fine-tunes an LLM directly on (preferred, rejected) pairs without training a separate reward model or running RL. It is a simpler, more stable alternative to RLHF.
When should I use Constitutional AI vs Direct Preference Optimization?
Constitutional AI is the right concept when you are focused on safety & alignment. When you have preference data and want a simpler pipeline than full RLHF.
Are Constitutional AI and Direct Preference Optimization the same thing?
No. Constitutional AI is safety & alignment; Direct Preference Optimization is training. They are related but address different parts of the AI stack.