Comparison
Positional Encoding vs Rotary Position Embedding
Positional Encoding and Rotary Position Embedding are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for Positional Encoding
Positional Encoding comes up when the question is fundamentally about architecture.
Adding a sine-wave pattern to each token by position.
When you would reach for Rotary Position Embedding
Rotary Position Embedding comes up when the question is fundamentally about architecture.
Llama 3 uses RoPE with adjustable base frequency.
Frequently asked
What is the difference between Positional Encoding and Rotary Position Embedding?
Positional Encoding: Positional encoding tells the transformer where each token sits in the sequence. Without it, "dog bites man" and "man bites dog" would look identical to the model. Rotary Position Embedding: RoPE encodes token position by rotating the query and key vectors in attention by an angle proportional to their position. It generalizes well to longer sequences than the model was trained on.
When should I use Positional Encoding vs Rotary Position Embedding?
Positional Encoding is the right concept when you are focused on architecture. Rotary Position Embedding applies when you are focused on architecture.
Are Positional Encoding and Rotary Position Embedding the same thing?
No. Positional Encoding is architecture; Rotary Position Embedding is architecture. They are related but address different parts of the AI stack.