Comparison
Pretraining vs TPU
Pretraining and TPU are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for Pretraining
Pretraining comes up when the question is fundamentally about training.
GPT-3 pretrained on ~300B tokens.
When you would reach for TPU
TPU comes up when the question is fundamentally about infrastructure.
Gemini trained on TPU v5p pods.
Frequently asked
What is the difference between Pretraining and TPU?
Pretraining: Pretraining is the initial training phase where an LLM learns to predict the next token on trillions of tokens of general text. It produces a base model that can be adapted later. TPU: TPUs are Google's custom AI accelerators, designed specifically for the matrix and reduction operations of neural networks. Used to train Gemini and large parts of Google's AI stack.
When should I use Pretraining vs TPU?
Pretraining is the right concept when you are focused on training. TPU applies when you are focused on infrastructure.
Are Pretraining and TPU the same thing?
No. Pretraining is training; TPU is infrastructure. They are related but address different parts of the AI stack.