Force-remove a compromised OAuth signing key

3 minute read

If an OAuth signing key on the operations center’s authorization server may be compromised, follow these steps to remove it immediately, without waiting for scheduled key rotation to complete. For example, $JENKINS_HOME/secrets/master.key on the operations center may have leaked through a host compromise, a backup leak, or a container escape, because that key decrypts the signing key at rest.

master.key is a product-wide CloudBees CI root secret. Protecting it through host hardening, backup handling, and container isolation is outside the scope of this procedure and of the CloudBees CI MCP Router. This procedure covers only the OAuth signing key.

Signing key rotation

The authorization server holds at most two signing keys:

  • Active key: Signs new tokens.

  • Retiring key: Stays published during a short overlap window so that tokens issued before a rotation still validate.

The key material is stored, encrypted, in $JENKINS_HOME/cloudbees-oauth-server/signing-keys.xml on the operations center.

Two behaviors make incident response necessary:

  • The operations center publishes public keys from the JSON Web Key Set (JWKS) endpoint (/oauth-server/jwks), and controllers cache the key set with a lifetime of up to 300 seconds (five minutes).

  • Scheduled rotation only drops the retiring key after the overlap window elapses. It does not evict a key on demand.

A single rotation does not remove a compromised key. Rotating once demotes the current active key to the retiring slot, where it stays published in the JWKS and continues to validate forged tokens. You must rotate twice to evict it; the second rotation overwrites the retiring slot with a freshly generated key.

Remove the compromised key

The following steps rotate the signing key twice, which evicts the compromised key from the JWKS regardless of whether it was the active or retiring key.

You need Overall/Administer permission on the operations center to complete this procedure.

To remove the compromised key:

  1. Fetch the JWKS from the authorization server and note the kid values currently published:

    curl -s https://oc.example.com/oauth-server/jwks | jq '.keys[].kid'
    ▼

    The active kid is also written to the operations center log on every rotation (Signing key rotated. Active kid=…​). Confirm which kid corresponds to the compromised key before continuing.

  2. On the operations center, navigate to Manage Jenkins  Script Console and run the following script. It rotates twice, which evicts the compromised key regardless of whether it was the active or the retiring key:

    import com.cloudbees.jenkins.plugins.oauth.server.SigningKeys

def keys = SigningKeys.get()
println "before: " + keys.allPublicJwks()*.getId()

// First rotation: the compromised active key becomes the retiring key (still published),
// and a fresh active key is generated.
keys.rotate()
// Second rotation: overwrites the retiring slot, evicting the compromised key entirely.
keys.rotate()

println "after:  " + keys.allPublicJwks()*.getId()
    ▼

    The compromised kid must not appear in the after: list. Both listed keys should be freshly generated.

    If the operations center Script Console is unavailable, stop the operations center, delete $JENKINS_HOME/cloudbees-oauth-server/signing-keys.xml, and restart. A new active key is generated on startup. This invalidates all previously issued tokens, not only forged ones.

  3. Fetch the JWKS again and confirm the compromised kid no longer appears:

    curl -s https://oc.example.com/oauth-server/jwks | jq '.keys[].kid'
    ▼

    New tokens are already signed with the fresh active key. Any token still bearing the compromised kid now fails validation at the authorization server with an unknown kid error.

Key rotation propagation

The authorization server stops advertising the compromised key immediately. Each controller caches the JWKS for up to 300 seconds (five minutes), so the change propagates across the fleet within one cache lifetime. A controller that receives a token with an unrecognized kid refreshes its cached key set sooner (subject to a short refetch cooldown), so in practice most controllers pick up the new key set within seconds of the next request.

Post-incident follow-up

After removing the key, take the following steps to investigate and prevent future exposure.

  • Investigate what exposed the key material. If master.key may have leaked, treat it as a CloudBees CI-wide incident.

  • Revoke affected users' grants and refresh tokens if their credentials may also be compromised.

For other OAuth issues, refer to Troubleshoot CloudBees CI MCP Router.

Introduction