Comparison
FlashAttention vs Sliding-Window Attention
FlashAttention and Sliding-Window Attention are both common AI/LLM terms but cover different ideas. Here is a quick side-by-side.
When you would reach for FlashAttention
FlashAttention comes up when the question is fundamentally about architecture.
Training a 70B model on 8K context that would not fit with standard attention.
When you would reach for Sliding-Window Attention
Sliding-Window Attention comes up when the question is fundamentally about architecture.
Mistral 7B: sliding window of 4096 over a 32K-trained context.
Frequently asked
What is the difference between FlashAttention and Sliding-Window Attention?
FlashAttention: FlashAttention is an algorithm that computes exact attention faster and with much less memory by carefully tiling the computation to fit in GPU SRAM rather than going to HBM. Sliding-Window Attention: Sliding-window attention limits each token to attending only the most recent W tokens (e.g. 4K), making attention linear in sequence length. Mistral and Gemma use it.
When should I use FlashAttention vs Sliding-Window Attention?
FlashAttention is the right concept when you are focused on architecture. Sliding-Window Attention applies when you are focused on architecture.
Are FlashAttention and Sliding-Window Attention the same thing?
No. FlashAttention is architecture; Sliding-Window Attention is architecture. They are related but address different parts of the AI stack.