Comparison
Inference vs Tensor Parallelism
Inference and Tensor Parallelism are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for Inference
Inference comes up when the question is fundamentally about inference.
A ChatGPT response: one inference call per turn.
When you would reach for Tensor Parallelism
Tensor Parallelism comes up when the question is fundamentally about infrastructure.
Llama 3 70B in BF16 (~140 GB) split across 4× H100 (80 GB each) with TP=4.
Frequently asked
What is the difference between Inference and Tensor Parallelism?
Inference: Inference is what happens when you actually run a trained model on new input. For LLMs that means generating tokens one at a time, with sampling and a KV cache. Tensor Parallelism: Tensor parallelism shards individual layers across multiple GPUs — splitting each matrix multiplication so different GPUs compute different output dimensions in parallel.
When should I use Inference vs Tensor Parallelism?
Inference is the right concept when you are focused on inference. Tensor Parallelism applies when you are focused on infrastructure.
Are Inference and Tensor Parallelism the same thing?
No. Inference is inference; Tensor Parallelism is infrastructure. They are related but address different parts of the AI stack.