Comparison
Speculative Decoding vs Time per Output Token
Speculative Decoding and Time per Output Token are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for Speculative Decoding
Speculative Decoding comes up when the question is fundamentally about inference.
Llama 3 70B accelerated by Llama 3 8B as draft.
When you would reach for Time per Output Token
Time per Output Token comes up when the question is fundamentally about inference.
A 70B model on H100: TPOT ~25ms (~40 tokens/sec).
Frequently asked
What is the difference between Speculative Decoding and Time per Output Token?
Speculative Decoding: Speculative decoding speeds up generation by having a small "draft" model propose several tokens, then verifying them in a single batched call to the big model. Time per Output Token: Time per output token (TPOT) is the average wall-clock delay between consecutive generated tokens during streaming. Determines how fast text appears once generation starts.
When should I use Speculative Decoding vs Time per Output Token?
Speculative Decoding is the right concept when you are focused on inference. Time per Output Token applies when you are focused on inference.
Are Speculative Decoding and Time per Output Token the same thing?
No. Speculative Decoding is inference; Time per Output Token is inference. They are related but address different parts of the AI stack.