Comparison
Inference vs Time per Output Token
Inference and Time per Output Token are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for Inference
Inference comes up when the question is fundamentally about inference.
A ChatGPT response: one inference call per turn.
When you would reach for Time per Output Token
Time per Output Token comes up when the question is fundamentally about inference.
A 70B model on H100: TPOT ~25ms (~40 tokens/sec).
Frequently asked
What is the difference between Inference and Time per Output Token?
Inference: Inference is what happens when you actually run a trained model on new input. For LLMs that means generating tokens one at a time, with sampling and a KV cache. Time per Output Token: Time per output token (TPOT) is the average wall-clock delay between consecutive generated tokens during streaming. Determines how fast text appears once generation starts.
When should I use Inference vs Time per Output Token?
Inference is the right concept when you are focused on inference. Time per Output Token applies when you are focused on inference.
Are Inference and Time per Output Token the same thing?
No. Inference is inference; Time per Output Token is inference. They are related but address different parts of the AI stack.