Comparison
Perplexity vs Pretraining
Perplexity and Pretraining are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for Perplexity
Perplexity comes up when the question is fundamentally about evaluation.
Perplexity 12 on WikiText is much better than perplexity 30.
When you would reach for Pretraining
Pretraining comes up when the question is fundamentally about training.
GPT-3 pretrained on ~300B tokens.
Frequently asked
What is the difference between Perplexity and Pretraining?
Perplexity: Perplexity measures how "surprised" a language model is by held-out text. Lower is better. It is the natural intrinsic eval for next-token prediction. Pretraining: Pretraining is the initial training phase where an LLM learns to predict the next token on trillions of tokens of general text. It produces a base model that can be adapted later.
When should I use Perplexity vs Pretraining?
Perplexity is the right concept when you are focused on evaluation. Pretraining applies when you are focused on training.
Are Perplexity and Pretraining the same thing?
No. Perplexity is evaluation; Pretraining is training. They are related but address different parts of the AI stack.