Sign everything, and plan for the day the key leaks
Commit, artifact and image signing is easy to switch on. Key custody and rotation are where teams get burned.
Turning on signing takes an afternoon. Deciding what happens when a signing key is compromised takes a lot more thought, and most teams skip it. Then the key leaks, and they learn that every verifier in the fleet trusts it, nobody knows where the copies are, and revoking it breaks production.
Three keys, three jobs
- Commit signing proves who wrote the change. Each engineer holds their own key, ideally on a hardware token, and the server rejects unsigned pushes to protected branches.
- Artifact signing proves the release came from your release process. That key never lives in a CI variable.
- Image signing proves what’s allowed to run. Admission control in the cluster rejects anything unsigned or signed by the wrong identity.
Mixing these up is the first mistake. If the key that signs container images is also an engineer’s personal key, a stolen laptop becomes a way to deploy anything.
Where the release key lives
The release key belongs in an HSM or a cloud KMS, with signing done by a call to that service. The pipeline gets permission to request a signature for one job on one protected branch. It never gets the key material. If a runner is compromised, the attacker can request signatures while they hold the runner, which is bad, but they cannot walk away with the key and sign things for the next year.
Keyless signing goes a step further. With Sigstore, the signer proves an identity through OIDC, gets a certificate that lives for minutes, and the signature is recorded in a public transparency log. There is no long-lived key to leak, and the log gives you a record of every signature made under your identity. Verifiers then check who signed, so write those identity rules carefully: the exact workflow and branch, not the whole organization.
The rotation drill
Write down, before you need it:
- Who can revoke each key, and how long it takes.
- Which verifiers need the new public key or identity rule, and how it gets to them.
- Which already-signed artifacts must be re-signed or pulled.
- How you will tell signatures made before the compromise from those made after.
The fourth point is why timestamps matter. If signatures carry a trusted timestamp, or sit in a transparency log, you can keep trusting artifacts signed before the compromise window and reject the rest. Without them, a leaked key means distrusting everything it ever signed.
Then run the drill once. You’ll find the verifier that has the old key baked into an image nobody owns. Better to find it during a drill than during an incident.
Signals worth alerting on
- A signature in the transparency log under your identity that no pipeline run produced.
- A signing request from a branch or job that isn’t on the allow list.
- Admission control rejecting images in a namespace that has never seen a rejection.
Where this lands in the standards
NIST SP 800-218 PS.2 asks for a way to verify the integrity of each release, and NIST SP 800-57 covers how long a key should live and how to retire it. The DoD Container Hardening Guide expects images to be signed and verified before they run. Signing satisfies the first sentence of each. The rotation drill is what makes it hold up on a bad day.