Sparse attention essentially combines 3 types of attention optimizations:<p>1. Compression of the query input vectors to reduce the size of the KV cache<p>2. Selectively computing uncompressed attention on a subset of tokens based on the compressed blocks with the highest attention scores<p>3. Using sliding window for local attention at full resolution<p>> Both Full Attention and sparse attention models are pretrained on 270B tokens of 8k-length texts, followed by continued training and supervised fine-tuning on 32k-length texts with YaRN to achieve long-context adaptation. Both models are trained to full convergence to ensure fair comparison.<p>> our experiments adopt a backbone combining Grouped-Query Attention (GQA) and Mixture-of-Experts (MoE), featuring 27B total parameters with 3B active parameters<p>Evaluated on MMLU, MMLU-PRO, CMMLU, BBH, GSM8K, MATH, DROP, MBPP, and HumanEval. NSA outperforms full attention on 7/9.<p>Beats out H2O, InfLLM, Quest, Exact-Top, and full attention on LongBench<p>Perfect retrieval on 64k needle-in-a-haystack<p>The CoT eval is less convincing, but outperforms the FA on AIME24.<p>Training speed of 2-9x vs. FlashAttention<p>Decoding speedup of 4-12x vs. full attention ["expected"? Didn't see comparison to other attention mechanisms]