Comparison
Foundation Model vs Pretraining
Foundation Model and Pretraining are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for Foundation Model
Foundation Model comes up when the question is fundamentally about foundations.
GPT-4 used by tens of thousands of applications via API.
When you would reach for Pretraining
Pretraining comes up when the question is fundamentally about training.
GPT-3 pretrained on ~300B tokens.
Frequently asked
What is the difference between Foundation Model and Pretraining?
Foundation Model: A foundation model is a single large model pretrained on broad data that can be adapted to many downstream tasks. LLMs are the most common type. Pretraining: Pretraining is the initial training phase where an LLM learns to predict the next token on trillions of tokens of general text. It produces a base model that can be adapted later.
When should I use Foundation Model vs Pretraining?
Foundation Model is the right concept when you are focused on foundations. Pretraining applies when you are focused on training.
Are Foundation Model and Pretraining the same thing?
No. Foundation Model is foundations; Pretraining is training. They are related but address different parts of the AI stack.