Container Image Signing with Cosign and Notary, Verified at Deploy Time

The gap that signing closes

Your pipeline builds an image, scans it, pushes it, and a cluster pulls it by tag. Between push and pull, a lot has to be true: the registry was not compromised, credentials with push access were not stolen, and nobody overwrote the tag. Most of the time nothing is wrong. But recent supply chain incidents have been about exactly these gaps.

Signing closes one of them. A signature produced by your pipeline and checked by your cluster means “this exact image was built by us”. It is a focused control, and it belongs alongside vulnerability scanning and hardening in our Docker containerisation service.

What good looks like

• Every production image is signed by the CI pipeline, never by a laptop.
• Signatures refer to the image digest.
• The cluster verifies signatures and the signer identity before admitting pods.
• SBOM and scan results are attached as signed attestations.
• Signing credentials, if any, are held in a KMS or hardware-backed store.
• There is a tested break-glass process.

Cosign keyless in practice

Keyless signing is the simplest secure option when you build in GitHub Actions or GitLab CI. The workflow gets an OIDC token, Sigstore’s certificate authority issues a short-lived certificate for that identity, and the signature is logged publicly. A sketch for GitHub Actions:

permissions:
  contents: read
  id-token: write   # required for keyless OIDC signing
  packages: write

steps:
  - uses: sigstore/cosign-installer@v3
  - name: Build and push
    id: build
    uses: docker/build-push-action@v6
    with:
      push: true
      tags: registry.example.com/platform/api:${{ github.sha }}
  - name: Sign by digest
    run: |
      cosign sign --yes \
        registry.example.com/platform/api@${{ steps.build.outputs.digest }}

Verification checks both who signed and which issuer vouched for them:

cosign verify \
  --certificate-identity-regexp '^https://github.com/acme/api/\.github/workflows/release\.yml@refs/heads/main$' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  registry.example.com/platform/api@sha256:<digest>

The identity pattern is the security control. A loose pattern such as “any workflow in any repository” would accept signatures from outside your organisation, so we write it narrowly and review it.

Enforcing at the cluster

A signature nobody checks adds nothing. In Kubernetes, an admission controller verifies images before pods run. Kyverno’s verifyImages rules and the Sigstore policy-controller are common choices; Gatekeeper with external data is another. A policy states which registries and which signer identities are acceptable. We roll out in audit mode first, review what would have been blocked, fix the gaps, and only then switch to enforcement. Policies apply to all namespaces you control, with documented exclusions for system components.

Keyless, KMS key or Notation?

Air-gapped or highly regulated environments may prefer KMS keys or Notation with an internal certificate authority. Teams on public CI with modest compliance needs usually do well with keyless.

Attestations

Beyond a signature on the image, you can attach signed statements about it, such as an SBOM from Syft or a record of the build provenance. This lets a policy ask more than “was it signed?”: it can ask “was it built from the main branch, and does it have an SBOM?” We add this once basic signing and verification are stable, not before.

Decisions and trade-offs

Should you do this now? Signing pays off when you have multiple people or systems with registry access, a compliance driver, or a customer requirement. For a three-person team with one registry and tight access, better wins come first: scanning, hardening and access control. We will tell you if signing can wait.

Third-party images. You cannot make upstream vendors sign for you. A common pattern is to mirror and scan third-party images, then re-sign them with your own identity once approved.

Operational cost. Signing adds a pipeline step and an admission dependency. Failing closed on a policy outage can block deploys, so we design for it.

Common failure modes

• Signing tags instead of digests, allowing the tag to be re-pointed.
• Policies that verify existence but not identity.
• Breaking cluster add-ons that pull unsigned system images. We exclude or mirror them deliberately.
• No rollback path when the policy engine misbehaves.
• Treating the signature as a security guarantee rather than provenance evidence.

Rollout plan that does not break deployments

The riskiest moment is the day enforcement switches on. We sequence it so nothing surprising happens:

1. Sign new builds from the pipeline for a couple of weeks before any policy exists, so there is a population of signed images.
2. Run the policy in audit mode and read the report. Expect to find system add-ons, operators and old workloads pulled from public registries.
3. Mirror or exclude each of those deliberately, with a named owner.
4. Enforce in a non-production cluster first and run the negative tests there.
5. Enforce in production namespace by namespace, starting with the least critical.

Rollback at each stage is a single policy change, which is why the policy lives in Git and goes through review like any other deployment. This pairs naturally with GitOps delivery, where policy changes are visible and auditable.

Verifying a signature by hand

During an incident, engineers should be able to answer “is this image really ours?” without waiting for a specialist. We include a short runbook command that prints the signer identity and issuer, so the answer is a copy and paste away. Knowing the verification command also makes the policy easier to trust, because the cluster is doing exactly what a person could do at a terminal.

Delivery and handover

We deliver the CI signing step, verification policies, tested negative cases and a runbook covering rotation, outages and exceptions, aligned with your registry setup and pipeline integration.

How to tell it is working

An unsigned image is rejected in a test namespace, a signed-by-the-wrong-identity image is also rejected, every production pod’s image can be traced to a signed build, and deploys are not slowed in any way users notice. To talk through your supply chain, get in touch.

References

• Overview of signing, Sigstore Cosign
• Verifying Signatures, Sigstore Cosign
• Verify Images, Kyverno documentation
• Notary Project documentation, Notary Project