Key Rotation Is Two Overlaps, Not One Switch

Rotating a JSON Web Token signing key looks like three steps: make a new key, start signing with it, delete the old one. Run those three steps in order and you will return 401 to a fraction of your traffic for a window you did not choose. The reason is that no verifier holds your keys — it holds a copy of your key set from whenever it last asked, and it validates against that copy.

A JWKS — a JSON Web Key Set, the document at your jwks_uri listing the public keys anyone may use to check your signatures — is fetched over HTTP and cached. In the .NET identity model libraries that cache lasts 12 hours by default. So there are two separate overlaps to get right, and they have different lengths:

• the new key has to be published long enough before its first use that every cache in the world has seen it;
• the old key has to stay published long enough after its last use that every token it signed has expired.

The panel below runs 48 hours of a fleet of 40 verifier instances, each with its own staggered cache, taking 2,400 requests a minute between them. Key B is published at hour 0. You choose when signing switches to it and when key A is deleted. The chart counts every validation that failed and the log says which of the two overlaps was too short.

Do three things in order. One: read the defaults — the new key goes into service the instant it is published, and 772,254 requests are rejected over 11 hours 41 minutes, peaking at 90% of all traffic. Two: drag publish → first use to 12 hours. Zero. Three: put it back at 0 and set the verifier to re-read the key set on every unknown key id. Also zero — so re-reading looks like it has solved rotation.

Now, with that still selected, drag keep the old key from 1 hour to 0. The rejections come back — 70,860 of them — and this time nothing you change about the verifier removes them.

The arithmetic behind both lines in that log is arithmetic you can do on paper. The publish overlap has to be at least the verifier cache lifetime, because that is the longest anyone can go without noticing a new key. The retire overlap has to be at least the token lifetime, because that is the longest a token signed by the old key can still be presented. Neither number is in any specification. Both are in your configuration.

The whole protocol is one paragraph, and it omits the numbers

OpenID Connect Core §10.1.1 is the specification of key rotation, and it is a single paragraph. Compressed, it says: publish your keys in a JWK Set at jwks_uri; put the key's identifier in the token header so the verifier knows which one to use; add new keys periodically; begin using a new one "at its discretion"; the verifier "knows to go back to the jwks_uri location to re-retrieve the keys when it sees an unfamiliar" key id; and the document "SHOULD retain recently decommissioned signing keys for a reasonable period of time to facilitate a smooth transition."

Two phrases in there are doing all the work, and neither is a number.

"At its discretion" is only true if the second phrase holds — if every verifier really does go back and look when it meets an unfamiliar key id. Set the verifier control to nothing and the panel shows what "at its discretion" costs when it does not: the signer's discretion becomes the verifiers' outage. The specification describes a pull-on-demand protocol, and a large share of deployed verifiers implement a fixed-lifetime cache instead.

"A reasonable period of time" is the second overlap, and it is the one with an exact answer. A token signed at the moment before you stopped using key A expires exactly one token lifetime later. Until then it is valid in every sense — signature correct, claims correct, in date — and the only thing that can reject it is the absence of its key. So "reasonable" means at least the maximum token lifetime you issue, plus whatever clock skew you allow. There is no shorter correct answer, and no verifier behaviour that rescues a shorter one.

Why re-reading fixes one half and not the other

Both halves produce the same log line. In PyJWT it reads Unable to find a signing key that matches: "…"; in node-jwks-rsa it is a SigningKeyNotFoundError; in the .NET libraries it is IDX10503: Signature validation failed. The token's kid is: '…', but did not match any keys. One message, two completely different situations, and the difference decides whether your incident is over in a minute or in eight hours. Only the .NET one prints the Keys tried list that tells you which situation you are in; with the other two you have to log it yourself.

Unknown key id, key is present in the document. The verifier's copy is stale. Re-reading resolves it, and every library that does so recovers on its own. This is the recoverable half, and it is what §10.1.1 is describing.

Unknown key id, key is absent from the document. The verifier's copy is fine. The key is gone. Re-reading returns the same document that already failed, and the request fails again. Nothing recovers until every affected token has expired — you cannot re-sign a token in someone else's hands, for the same reason you cannot recall one; see what already-issued tokens are immune to. Drag keep the old key to 0 with the verifier set to re-read on every unknown key id: 70,860 requests are still rejected, and the fleet is now making 2,160 key-set reads a minute — one per rejected request — to keep learning the same thing.

That second case is also the only reason to be careful about the emergency version of this operation. Deleting a compromised key immediately is correct and it will reject every unexpired token signed with it. That is the point. What matters is knowing, before you do it, that the blast radius is one token lifetime of your entire estate, and that nothing else in the validation pipeline will soften it.

Three libraries, three different rotations

The same rotation schedule is safe or catastrophic depending on which verifier is reading it, and the differences are in default arguments rather than in documentation.

• Microsoft.IdentityModel (.NET) caches the whole discovery configuration, key set included. DefaultAutomaticRefreshInterval is 12 hours; an out-of-band refresh triggered by a validation failure is rate-limited by DefaultRefreshInterval, 5 minutes. So a new key can cost up to five minutes of rejections per instance, and a fleet that restarts rarely can be running on a twelve-hour-old key set.
• node-jwks-rsa memoises per key id, not per document, with cacheMaxAge of 600,000 ms — ten minutes. An unfamiliar key id is a cache miss, so it reads immediately. Rotation is nearly free here, which is why the library also ships jwksRequestsPerMinute, defaulting to 10 when rate limiting is switched on, and raising a rate-limit error past it.
• PyJWT's PyJWKClient caches the key set for lifespan, 300 seconds, and on an unmatched key id it refreshes and retries once — with no throttle of any kind. Rotation is free; a burst of tokens carrying a key id you never published is a burst of synchronous HTTP requests from your API to your identity provider.

Read that list as one sentence: the safe publish lead time for your system is the largest cache lifetime among all your verifiers, and you do not control most of them. Partners, mobile clients, sidecars, gateways and that one service still on a framework version from three years ago all get a vote. Twenty-four hours of lead is not conservative; it is roughly two of .NET's cache intervals, which is the smallest number that survives an instance refreshing one minute before you published.

The schedule, written down

Everything above collapses into four timestamps. Set them, in this order, and the rotation is invisible.

1. T₀ — publish. Add the new key to the JWK Set. Sign nothing with it. The document now has both keys, which is legal, ordinary and how the set is meant to look most of the time.
2. T₀ + max verifier cache — switch. Start signing with the new key and stop using the old one. Use the largest cache lifetime you can identify, then double it. In the panel, any lead at or beyond the cache lifetime drives the publish half of the failure count to zero even with re-reading turned off entirely.
3. Switch + max token lifetime + skew — retire. Remove the old key. Take the lifetime from a real token's exp minus its iat, not from configuration, and add whatever clock skew your verifiers tolerate.
4. Then keep going. Publish the next key at retire time. Steady state is two keys in the set at all times, one signing and one warming up, and rotation stops being an event you schedule an outage for.

Two things to check before you trust the schedule. First, that your key set is served with cache headers you chose. OpenID Connect Core §10.2.1 says the jwks_uri response SHOULD carry a Cache-Control header with a max-age directive, and if the document has none an intermediate proxy will invent one — at which point the cache you have to beat is no longer the one in the library. Second, that the token header's key id is actually present and matches the key set's entries — the identifier is a hint the verifier looks up, and a signer that omits it forces every verifier to try every key, which works right up until the moment the set has two keys of the same type.

Measuring your own two overlaps

Both numbers are observable today, and neither requires the identity provider's cooperation.

• Your retire overlap is one subtraction. Decode a real token from production and compute exp − iat — the expiry claim minus the issued-at claim, both Unix timestamps in seconds. That is the minimum number of seconds the old key must remain published after its last use. If your provider issues several token types, take the longest — refresh tokens and identity tokens are often signed by the same key with very different lifetimes.
• Your publish overlap is a grep. Search your services for the cache settings by name: AutomaticRefreshInterval, cacheMaxAge, lifespan, jwks-refresh-interval. Anything you do not find is running on the library default, which is the value in the list above and is usually the largest one in your estate.
• Look at the key set right now. curl -s $ISSUER/.well-known/openid-configuration, take jwks_uri, and fetch it. Count the keys. One key means you have never rotated, or you rotate by cutover and have simply been lucky. Two or three with different identifiers means someone set this up properly.
• Look at the response headers on that same request. curl -sI the jwks_uri. Whatever Cache-Control: max-age says is what every proxy between you and the provider will honour, and it stacks on top of the library cache rather than replacing it.
• Instrument the two cases separately. When your verifier raises a key-not-found, log the key id from the token header and the list of key ids in the copy it was holding, plus that copy's age. Those three values tell you in one line whether you are stale or whether the key is gone, and they are the difference between waiting a minute and declaring an incident.
• Rehearse it in staging with the clock. Publish a key, wait, switch, wait, retire — and take a 401 count at each step. A rotation that has never been run is a runbook, not a capability, and the first time you need it will be the day a key leaked.

The same overlap shows up wherever one party publishes a key and another caches it. Pinning a public key in a mobile client is this problem with a worse failure mode, because the stale copy is on a device you cannot reach — see why a pin set needs a backup pin. And the reason anyone rotates in the first place sits next door in what a private key at the other end of a connection is actually proving.