Llm Linear Attention

Where this is already explained

• Scaled Dot-Product Attention

  That attention is O(n²) full stop, so each generated token costs quadratically more as context grows, and that the quadratic term is what makes long context expensive. Both halves are wrong. The n² comes from running n queries against n keys in a single pass, which only happens during prefill; a decode step has exactly one query, so it does n dot products, not n² — linear per token. And even in prefill the quadratic attention term does not overtake the linear-in-n weight matmuls until roughly 26,000 tokens on an 8B model, so at ordinary prompt lengths attention is a minority of the arithmetic. People also read the causal mask as an optimisation that halves work, when what it actually buys is the guarantee that row i of a parallel prefill is identical to what a sequential decode would have produced at step i.

• FlashAttention

  That FlashAttention is a faster approximation of attention, in the same family as Linformer or Performer, so it trades a little accuracy for speed. It computes exactly the function standard attention computes, and it does so while performing slightly more arithmetic, not less. The speedup comes entirely from bytes not moved between HBM and the chip. The second half of the misconception is that it shrinks the KV cache: it does not touch the cache, only the N x N intermediate, so it buys long prompts and nothing at all in the memory ledger of decoding.

• num_heads Is Not a Capacity Knob

  That the number of heads is a size setting: more heads means more parameters, more arithmetic, and a more expressive layer, so raising num_attention_heads is a way to make the model stronger. Nothing about the cost changes. With d_k = d_v = d_model/h the four projection matrices together are 4 x d_model^2 weights at every head count, and the score and value matmuls do 2 x n x d_model multiply-accumulates per token at every head count. The one thing that does change is head_dim, and head_dim is a hard ceiling on the rank of that head's n-by-n score matrix — so heads trade the complexity of a single pattern for the number of patterns the layer can run at once.

• Raising max_position_embeddings Does Not Extend Context

  That a model's context length is the value of max_position_embeddings, so raising it extends the window. RoPE has no per-position parameters to run out of; the limit is that roughly a quarter to a half of the dimension pairs never complete a single full rotation inside the training window, so at any longer position they are rotated to angles the model has never been asked to interpret. For Llama 2 7B that is 18 of the 64 pairs. The scaling methods are not conveniences — position interpolation removes every unseen angle but shrinks the separation between adjacent positions to 0.1% of what it was at a scale factor of 32, and YaRN exists because those two failures live in different halves of the frequency ladder.