Comparison
Fine-tuning vs HumanEval
Fine-tuning and HumanEval are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for Fine-tuning
After you've exhausted prompting and retrieval, and you have a few hundred to thousands of clean labeled examples.
Fine-tuning Llama 3 on medical Q&A for a clinical assistant.
When you would reach for HumanEval
HumanEval comes up when the question is fundamentally about evaluation.
GPT-4: ~88% pass@1 on HumanEval.
Frequently asked
What is the difference between Fine-tuning and HumanEval?
Fine-tuning: Fine-tuning continues training a pretrained model on a smaller, task-specific dataset, adjusting its weights to specialize behavior or knowledge. HumanEval: HumanEval is a benchmark of 164 hand-written Python programming problems, each with a function signature, docstring, and unit tests. The model writes the function body.
When should I use Fine-tuning vs HumanEval?
After you've exhausted prompting and retrieval, and you have a few hundred to thousands of clean labeled examples. HumanEval applies when you are focused on evaluation.
Are Fine-tuning and HumanEval the same thing?
No. Fine-tuning is training; HumanEval is evaluation. They are related but address different parts of the AI stack.