Comparison
MMLU vs Reference-Based Evaluation
MMLU and Reference-Based Evaluation are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for MMLU
MMLU comes up when the question is fundamentally about evaluation.
GPT-4: 86.4% MMLU (5-shot, original release).
When you would reach for Reference-Based Evaluation
When ground truth is available and one or a small number of outputs are clearly correct — extraction, classification, structured-output, code-with-tests.
A classifier eval: gold label is "spam", model output is "spam" → match, score 1.
Frequently asked
What is the difference between MMLU and Reference-Based Evaluation?
MMLU: MMLU is a benchmark of ~16K multiple-choice questions across 57 subjects from elementary to professional. It is one of the most widely cited LLM benchmarks. Reference-Based Evaluation: Reference-based evaluation compares the model output against a known correct answer using exact match, edit distance, BLEU, ROUGE, or LLM-as-judge "matches the reference."
When should I use MMLU vs Reference-Based Evaluation?
MMLU is the right concept when you are focused on evaluation. When ground truth is available and one or a small number of outputs are clearly correct — extraction, classification, structured-output, code-with-tests.
Are MMLU and Reference-Based Evaluation the same thing?
No. MMLU is evaluation; Reference-Based Evaluation is evaluation. They are related but address different parts of the AI stack.