Raising max_position_embeddings Does Not Extend Context

A model's context limit is not stored anywhere. Rotary position embedding — RoPE, the scheme almost every current open-weight model uses — has no per-position parameters to run out of. What runs out is experience: each pair of dimensions inside a head spins at its own speed, and the slow ones never finish a single turn during training. Ask them about position 100,000 and they will hand back an angle the model has never had to interpret.

Here is the whole mechanism in two lines. A head of size d — the head_dim that the head count decides — is cut into d/2 pairs of dimensions. Pair j gets a frequency θ_j = base−2j/d, where base is the rope_theta in your config. A token at position m has its query and key rotated within pair j by the angle m·θ_j. Because rotating both by the same amount cancels, the dot product between two tokens depends only on the difference of their positions — that is the property the RoPE paper was written to obtain.

So a dimension pair has a wavelength: the distance λ_j = 2π/θ_j over which it makes one complete turn. Pair 0 has a wavelength of about 6 tokens. Pair 63 has a wavelength of tens of thousands. The panel lays out that ladder and shows what each of the four things people do to it actually does.

Pick Llama 2 7B, drag the target context to 131,072, and leave the method on just raise the number.

Eighteen of the sixty-four pairs come up magenta, and the worst-case excursion reads 32.0×. Read the last row of the third panel: pair 63 has been rotated through 27.1° in total across the entire training run — less than a twelfth of a circle, ever — and you have just asked it about 867.2°, which is nearly two and a half revolutions it has no experience of. Now switch to position interpolation. Every magenta bar goes green — no pair is asked anything new — and the adjacent-position separation readout falls to 0.1%. The first bar of the second chart goes to the top. The model has stopped being able to tell adjacent tokens apart.

Why there is nothing to run out of

The original Transformer added a fixed sinusoidal vector to the token embedding at the bottom of the stack. Vaswani et al. gave the reason in §3.5: We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset k, PE_pos+k can be represented as a linear function of PE_pos. A hypothesis is what it stayed. Adding a position vector to a content vector means the attention score mixes content-content, content-position and position-position terms, and nothing forces the result to depend only on the distance.

RoPE does not add anything. It rotates, and it rotates the query and the key inside the attention computation of every layer — not the embedding, and never the value. Pair j of a head is treated as a point in a plane, and a token at position m turns that point by m·θ_j. When the score for positions m and n is computed, the two rotations compose into a single rotation by (m−n)·θ_j. The absolute positions cancel. That is not a hypothesis; it is what a rotation matrix does, and it is the whole reason the paper exists.

The frequencies come from one line. vLLM's base implementation reads inv_freq = 1.0 / (base ** (arange(0, rotary_dim, 2) / rotary_dim)), which is θ_j = base−2j/d for j = 0, 1, … d/2−1. With d = 128 and base = 10000 the fastest pair turns once every 6.3 tokens and the slowest once every 54,410. There is no table of positions, no learned embedding matrix, nothing indexed by m. Nothing can overflow.

Which is exactly why the failure is invisible until you look at it the way the panel does. Feed position 100,000 into the formula and you get a perfectly well-formed rotation. It is a rotation the model has no experience of, and the model has no way to signal that.

Two different things you can break

Look at the frequency ladder for Llama 2. Pair 0 makes 651.9 turns inside the training window; pair 63 makes 0.075 of one. The boundary — the last pair that manages a full turn — is pair 45, which the panel rounds to 1.00 turns, with a wavelength of 4,080 tokens against a 4,096-token window. Below that line, 18 pairs have never seen a complete circle.

That number is a property of the model, not of what you do to it. Put the method back to just raise the number — the summary row only counts pairs for you there — and switch the model to Llama 3 8B: it becomes 29 of 64, because rope_theta went from 10,000 to 500,000 and every wavelength grew with it. This is the part that reads backwards at first: raising the base makes more pairs slow, not fewer. That is deliberate. A pair whose wavelength is shorter than the context wraps around, so it cannot by itself distinguish two tokens 5,000 apart from two tokens 5,000 − λ apart. The pairs with wavelengths longer than the whole window are the only ones that give an unambiguous long-range signal — and they are, necessarily, the same pairs that never complete a turn in training. The property you want at long range and the property that breaks under extrapolation are the same property.

So there are two distinct ways to be wrong, and each method picks one.

Leave the frequencies alone. Switch the model back to Llama 2 7B for the rest of this. Adjacent-position separation stays at 100% — nothing local is harmed at all — and 18 pairs are rotated past their arc, the worst by a factor of 32. The YaRN paper puts the consequence in its abstract without hedging: these models fail to generalize past the sequence length they were trained on.

Divide every frequency by the scale. Position interpolation, from Chen et al. Position 131,072 now lands where 4,096 used to, so the unseen count is 0 at every scale. The price is in the second chart: at scale 32 the adjacent-position separation is 0.1% of what it was, and the similarity between a token and its immediate neighbour is 1.000 to three decimal places. Positions 1, 4 and 16 apart have become nearly the same position. This is why the paper reports fine-tuning as part of the method rather than as an optional extra — within 1000 steps — and why an interpolated model can get worse at short prompts, where nothing needed extending in the first place.

Raise the base instead. The NTK-aware variant multiplies the base by sd/(d−2), which leaves pair 0 untouched and interpolates the last pair by almost exactly s. At scale 32 the readouts say 17 unseen pairs and 76% separation. It is a genuine compromise, and it is a compromise chosen by an exponent rather than by asking where the boundary actually is.

What YaRN is, exactly

Switch the method to YaRN at scale 32 and both readouts are good at once: 0 unseen pairs, 99.9% separation. That is not a better formula. It is the same two formulas, applied to different parts of the ladder, with the split decided by counting turns.

vLLM's implementation is short enough to quote in structure. It computes both candidate frequency sets — inv_freq_extrapolation, the original, and inv_freq_interpolation, the original divided by the scaling factor — then blends them per dimension with a mask from yarn_find_correction_range(beta_fast, beta_slow, …). The defaults in that file are beta_fast: int = 32 and beta_slow: int = 1, and the correction function is

dim · ln(L / (r · 2π)) / (2 · ln base)

which is exactly the pair index whose wavelength fits into the training window r times, solved for the index. So beta_slow = 1 is one full turn and beta_fast = 32 is thirty-two of them. For Llama 2 those come out at pair 20 and pair 46 — and the boundary the panel finds by direct measurement, the last pair that completes a turn, is pair 45. The threshold is not a tuned constant. It is the same line the ladder already had in it.

Everything at or below pair 20 keeps its original frequency: those pairs turn at least 32 times in training, so they have seen every angle and can be trusted to extrapolate. Everything at or above pair 46 is fully interpolated: those pairs have never seen a full circle, so they must be compressed into the arc they know. Between them is a linear ramp. That is the entire method, plus one correction: because interpolating frequencies changes the typical magnitude of attention scores, YaRN rescales by yarn_get_mscale(scale) = 0.1 · ln(scale) + 1.0, which at scale 32 is the 1.3466 in the fourth readout.

Where that factor is applied is worth being exact about, because the obvious reading of it is wrong. vLLM does not multiply the logits by 1.3466. It multiplies the rotation tables — cos = freqs.cos() * self.mscale and the same for sin — so the factor is carried by the rotated query and by the rotated key, and the dot product between them comes out multiplied by mscale², which is 1.8133 at scale 32. The squaring is not an accident. YaRN defines an attention temperature t and multiplies logits by 1/t; what it gives you a formula for is √(1/t), and that is the number in the readout. Scaling the embeddings by √(1/t) is how you get 1/t onto the logit without touching the softmax at all.

None of this makes the extension free. YaRN is a re-parameterisation that keeps the model's existing angles meaningful; the paper still trains, and reports needing 10x less tokens and 2.5x less training steps than previous methods — which is a comparison between amounts of training, not an absence of it. The panel shows the geometry, not the loss curve. A configuration with zero unseen pairs and full local separation is a configuration that could work, not one that does.

The boundary

Nothing here fixes attention itself. Every method in the panel changes which angles the rotation produces. None of them changes the fact that scoring 131,072 keys costs 32 times what scoring 4,096 does, or that the key-value cache for that context is 32 times larger. Extending the window is a positional problem and a memory problem, and solving the first one does not touch the second.

Dynamic scaling changes the answer per request. vLLM ships a dynamic_ntk_scaling_rope alongside the fixed one, which computes the scaling factor from the actual sequence length rather than from a configured maximum. That keeps short prompts at scale 1 and avoids the short-prompt regression, at the cost of the frequency table changing underneath a running sequence.

The whole family is one answer to the question, not the only one. Press, Smith and Lewis asked it directly in Train Short, Test Long: how does a model extrapolate at all? Their answer, ALiBi, adds a linear penalty to attention scores in proportion to distance and has no rotation and no frequencies to run out of. It is a different trade — a fixed recency bias in exchange for extrapolation — and it is worth knowing about precisely because it shows the problem this lesson describes is a property of RoPE's design rather than of transformers. A third answer leaves position alone and rewrites the score function instead, which is what linear attention does.

The simulation is geometry, not quality. It computes real frequencies, real angles and the real long-term-decay quantity, and it will tell you which dimension pairs are being asked something new. It cannot tell you whether a given model survives it, because that depends on training the panel knows nothing about. Treat a magenta bar as this needs fine-tuning or a different method, never as a prediction of perplexity.

Checking it in a real system

1. Read three fields, in this order. rope_theta, max_position_embeddings, and rope_scaling. The first two tell you where the one-turn boundary sits. The third tells you whether anybody has already done something about it. If rope_scaling is null and someone has raised max_position_embeddings, that is the configuration this whole lesson is about.

2. Find out what the model was actually trained on, not what its config says. When a scaling method is applied, the training length lives in rope_scaling.original_max_position_embeddings, and max_position_embeddings becomes the extended figure. The Megatron-LM issue titled Dual meaning of max_position_embeddings, computing both embedding shape & yarn scaling base exists because the same field is read for two purposes in the same codebase. If the two numbers disagree, the smaller one is the one your intuition should use.

3. Compute the boundary yourself; it is four lines. lambda_j = 2*pi*base**(2*j/d) for j in range(d//2), and count how many exceed the training length. That count is the number of dimension pairs with no experience of a full rotation, and it is the number that decides whether you can extrapolate at all. For a 128-dimensional head the answer is a single integer and takes a second to produce.

4. Test at the boundary, not in the middle. A model extended to 128k that is evaluated on 8k prompts is being asked nothing hard. Put the fact you need at position 120,000 and ask for it — the passkey-retrieval setup the position interpolation paper used — and separately re-run your short prompt evaluations, because interpolation degrades exactly there and nobody thinks to check.

5. If you must extend without training, prefer the method that respects the boundary. YaRN over NTK-aware over linear over nothing, and expect to fine-tune anyway. If the serving stack offers dynamic scaling and your traffic is mostly short, that is usually the cheaper trade than a fixed factor applied to every request.

6. Do not compare the base across models as if it were a quality setting. Llama 2's 10,000, Llama 3's 500,000 and Qwen3's 1,000,000 are each paired with a different training length, and it is the ratio that means anything. Switch the model in the panel and watch both the base and the trained window change together — the useful comparison is the shape of the ladder against the window, which is the picture, not either number alone.