Comparison
Deep Learning vs Pretraining
Deep Learning and Pretraining are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for Deep Learning
Deep Learning comes up when the question is fundamentally about foundations.
Image recognition models like ResNet.
When you would reach for Pretraining
Pretraining comes up when the question is fundamentally about training.
GPT-3 pretrained on ~300B tokens.
Frequently asked
What is the difference between Deep Learning and Pretraining?
Deep Learning: Deep learning is machine learning using neural networks with many layers ("deep" = many layers). It powers nearly every recent breakthrough in AI, including LLMs and image generators. Pretraining: Pretraining is the initial training phase where an LLM learns to predict the next token on trillions of tokens of general text. It produces a base model that can be adapted later.
When should I use Deep Learning vs Pretraining?
Deep Learning is the right concept when you are focused on foundations. Pretraining applies when you are focused on training.
Are Deep Learning and Pretraining the same thing?
No. Deep Learning is foundations; Pretraining is training. They are related but address different parts of the AI stack.