Foundations · intermediate
Foundation Model
A foundation model is a single large model pretrained on broad data that can be adapted to many downstream tasks. LLMs are the most common type.
Explanation
The term, coined by Stanford's CRFM in 2021, describes a shift in AI economics: instead of training a separate model per task, organizations train one very large general-purpose model and then fine-tune or prompt it for everything else.
Foundation models are usually trained on internet-scale data, cost millions of dollars to train, and are reused across thousands of applications. The same Claude or GPT model that answers customer support emails also writes code, summarizes meetings, and drafts marketing copy.
LLMs are the most common foundation models, but the term also covers multimodal models (vision + language), code models, and some image generators.
Examples
- GPT-4 used by tens of thousands of applications via API.
- Llama 3 as the foundation for many open-source fine-tunes.
- CLIP as a vision-language foundation model.
Frequently asked
What is Foundation Model?
A foundation model is a single large model pretrained on broad data that can be adapted to many downstream tasks. LLMs are the most common type.
What is an example of foundation model?
GPT-4 used by tens of thousands of applications via API.
How is Foundation Model related to Large Language Model?
Foundation Model and Large Language Model are both foundations concepts. A large language model is a neural network trained on huge amounts of text to predict the next token in a sequence. GPT-4, Claude, and Gemini are all LLMs.
Is Foundation Model considered intermediate?
Foundation Model is generally considered intermediate-level material in the AI and LLM space.