Comparison
Data Contamination vs Pretraining
Data Contamination and Pretraining are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for Data Contamination
Data Contamination comes up when the question is fundamentally about evaluation.
MMLU questions appearing verbatim in pretraining data crawls.
When you would reach for Pretraining
Pretraining comes up when the question is fundamentally about training.
GPT-3 pretrained on ~300B tokens.
Frequently asked
What is the difference between Data Contamination and Pretraining?
Data Contamination: Data contamination is when benchmark questions or answers leak into a model's pretraining corpus, inflating its score because it memorized rather than reasoned. Pretraining: Pretraining is the initial training phase where an LLM learns to predict the next token on trillions of tokens of general text. It produces a base model that can be adapted later.
When should I use Data Contamination vs Pretraining?
Data Contamination is the right concept when you are focused on evaluation. Pretraining applies when you are focused on training.
Are Data Contamination and Pretraining the same thing?
No. Data Contamination is evaluation; Pretraining is training. They are related but address different parts of the AI stack.