Comparison
Pretraining vs Reinforcement Learning from Human Feedback
Pretraining and Reinforcement Learning from Human Feedback are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for Pretraining
Pretraining comes up when the question is fundamentally about training.
GPT-3 pretrained on ~300B tokens.
When you would reach for Reinforcement Learning from Human Feedback
Reinforcement Learning from Human Feedback comes up when the question is fundamentally about training.
ChatGPT trained with RLHF to refuse unsafe requests.
Frequently asked
What is the difference between Pretraining and Reinforcement Learning from Human Feedback?
Pretraining: Pretraining is the initial training phase where an LLM learns to predict the next token on trillions of tokens of general text. It produces a base model that can be adapted later. Reinforcement Learning from Human Feedback: RLHF fine-tunes an LLM to maximize a reward model that was itself trained on human preference judgments between candidate responses.
When should I use Pretraining vs Reinforcement Learning from Human Feedback?
Pretraining is the right concept when you are focused on training. Reinforcement Learning from Human Feedback applies when you are focused on training.
Are Pretraining and Reinforcement Learning from Human Feedback the same thing?
No. Pretraining is training; Reinforcement Learning from Human Feedback is training. They are related but address different parts of the AI stack.