Comparison
Learning Rate vs Scaling Laws
Learning Rate and Scaling Laws are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for Learning Rate
Learning Rate comes up when the question is fundamentally about training.
Pretraining: peak LR around 1e-4 with cosine decay.
When you would reach for Scaling Laws
Scaling Laws comes up when the question is fundamentally about training.
Predicting GPT-4's loss before training based on smaller-scale runs.
Frequently asked
What is the difference between Learning Rate and Scaling Laws?
Learning Rate: The learning rate is the step size used to update weights during training. Too high and training diverges; too low and it crawls or gets stuck. Scaling Laws: Scaling laws are the empirical power-law relationship between model size, training data, training compute, and resulting loss. They predict that bigger, more data-fed models keep improving in a smooth, forecastable way.
When should I use Learning Rate vs Scaling Laws?
Learning Rate is the right concept when you are focused on training. Scaling Laws applies when you are focused on training.
Are Learning Rate and Scaling Laws the same thing?
No. Learning Rate is training; Scaling Laws is training. They are related but address different parts of the AI stack.