Kubernetes Supply Chain Security on AKS — the cargo nobody inspected

Five controls guard what the cluster does with a workload. None of them look inside it.

The ship is sound, the crew is vetted, the harbor is walled. The manifest is signed by a clerk who never opened a crate.

§IFrame

Count what the arc has built. Tuesday of the second sprint week gave the reconciliation loop, which watches desired state and drives the world toward it. Saturday gave the admission gate, which stops a Pod at the door if its security context is wrong. The next visit gave the Service, a stable name over churning endpoints. The visit after gave NetworkPolicy, which walls the flat network into allow-lists. Monday's visit gave RBAC and Workload Identity, which decide who the caller is and what the caller may touch.

Five controls. Every one of them governs a container that has already been chosen.

Ask the question the arc has been deferring. When the scheduler places a Pod and the kubelet pulls myapp:latest from a registry, what in that chain asserted that the bytes returned are the bytes someone built, reviewed, and approved? The RoleBinding did not. The NetworkPolicy did not. Pod Security Admission checked whether the container asks for privilege; it never checked what the container is.

This is the cargo nobody inspected.

Rice puts the count plainly in the chapter on software vulnerabilities: an image is a filesystem plus a configuration, and the filesystem carries every package the base image dragged along. A Pod running as non-root inside a locked namespace with a scoped ServiceAccount is still a Pod running whatever an unaudited FROM line pulled in eighteen months ago.

§IIFoundations: three facts about an image

Fact one. An image is content-addressable, and its usual name is not.

Poulton draws the distinction sharply in the digest section of Docker Deep Dive. A tag is mutable. v1.4.0 points wherever the last push pointed it; nothing in the registry protocol forbids re-pushing a different manifest under the same tag tomorrow. A digest is a SHA-256 hash of the image manifest. It cannot point elsewhere, because changing the content changes the digest.

Coin it and keep it: a tag is a promise, a digest is a fact. Rice makes the same distinction in the Identifying Images section by the effect it has on an audit. When an incident asks what was running on Tuesday, the tag answers with a name and the digest answers with the artifact.

Fact two. A scan is a claim with a timestamp on it.

Rice's chapter on vulnerabilities carries a caution most pipelines ignore. A scanner compares the packages installed in the image against a vulnerability database as it stood at scan time. Three things then rot that claim: new CVEs get published against packages that were clean, maintainers mark issues won't fix so the report never clears, and zero-days exist by definition before the database knows them.

So the scan has a timestamp. An image scanned clean at build and deployed unchanged for a year has not been clean for a year. It has been unexamined for a year. Rice's answer is regular re-scanning of what is deployed, not only of what is being built.

Fact three. The registry is a trust boundary, and most clusters do not treat it as one.

Rice's Storing Images section names the failure directly: a cluster that can pull from any registry on the internet has an unbounded set of suppliers. Azure Container Registry, ECR, GCR, and Artifact Registry all solve the same problem the same way. Give the workload identity a pull role on exactly one registry, then refuse everything else at admission.

§IIIMechanism: where the checks actually attach

Four attach points, in the order a build travels.

At authoring, the Dockerfile. Rice's best-practices section lists the moves that shrink the attack surface before any scanner runs: pin the base image, run as a named non-root user, use multi-stage builds so compilers and package caches never reach the final layer, avoid ADD with remote URLs, and never bake secrets into a layer. A deleted file in a later layer is still present in the earlier one; the layer is the unit of persistence, not the final filesystem.

At build, the pipeline. Poulton's real-world security chapter puts the scan inside CI with a policy that fails the build on HIGH and CRITICAL. Two artifacts come out of this stage. The scan report, and the SBOM: the list of every package and version the image contains, in SPDX or CycloneDX form. The SBOM matters because it answers the question a scan cannot. When a new CVE lands next month, which of the four hundred images already deployed contain the affected package? Grepping SBOMs answers that in seconds. Re-scanning four hundred images does not.

At push, the registry. ACR holds the image and can be configured to be the only registry the cluster trusts. Rice's Signing Images section covers the concept: a signature binds a digest to an identity, so a consumer can verify that a specific artifact came from a specific builder.

At admission, the cluster. This is the enforcement point, and the one the arc has already built machinery for. Burns and co-authors lay out the admission flow in the policy-and-governance chapter: a request passes authentication, then authorization, then the admission chain, and only then reaches etcd. An image policy is one more validating webhook in that chain. Kubernetes also ships a built-in ImagePolicyWebhook admission plugin, configured on the API server, which calls out to an external scanner service for a verdict on each image.

The four attach points form a rule worth stating once. Every check before admission is advice. Admission is the only one that is a decision. A pipeline scan that fails a build is a strong convention, and a determined engineer with registry push rights routes around it in a minute. The webhook does not care how the image got into the registry.

§IVWorked Example: the trusted-registry boundary on AKS

Take the trading namespace from the NetworkPolicy lesson and give it a supply-chain boundary.

Step one. Attach ACR so pull works without a stored credential.

az aks update \
  --resource-group hedronite-rg \
  --name hedronite-aks \
  --attach-acr hedroniteacr

This grants the cluster's kubelet identity the AcrPull role on that registry. No imagePullSecret lands in the namespace, which removes a long-lived credential of exactly the kind Monday's lesson argued against. The token is a lease held by the node identity, not a key sitting in a Secret.

Step two. Turn on the registry's own scanning and read its verdict.

az aks enable-addons \
  --resource-group hedronite-rg \
  --name hedronite-aks \
  --addons defender \
  --enable-defender

trivy image --severity HIGH,CRITICAL --exit-code 1 \
  hedroniteacr.azurecr.io/trading-gateway:1.4.0

The first command turns on continuous assessment of what is running. The second is the CI gate, and its --exit-code 1 is what makes it a gate rather than a report. A scan that prints findings and exits zero is a log line.

Step three. Resolve the tag to a digest and deploy the digest.

az acr manifest show-metadata \
  --registry hedroniteacr \
  --name trading-gateway:1.4.0 \
  --query digest -o tsv

The output is a sha256: string. The Deployment then references the image by digest rather than tag:

spec:
  containers:
    - name: gateway
      image: hedroniteacr.azurecr.io/trading-gateway@sha256:6c3e0b...

What this buys: the thing that was scanned in step two is provably the thing that runs. With a tag, those are two different questions with a window between them. This is where the digest stops being trivia and starts being control.

Step four. Refuse everything else at the door.

A Gatekeeper or Kyverno policy, or a custom validating webhook, enforces two rules across the namespace. First, the image reference must begin with hedroniteacr.azurecr.io/. Second, the reference must contain an @sha256: digest rather than a tag.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
  name: trading-registry-boundary
spec:
  match:
    namespaces: ["trading"]
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    repos:
      - "hedroniteacr.azurecr.io/"

§VConnection to Prior Lessons

The arc's shape is now visible. The reconciliation loop asked what should exist. Admission asked may this run. Services asked what do we call it. NetworkPolicy asked who may reach it. RBAC and Workload Identity asked who is calling. This lesson asks what is it made of, and the answer lands in the same enforcement seam the arc has been building since the second sprint week: the admission chain.

Monday's lesson coined the token as a lease rather than a key. The --attach-acr binding here is the same idea moved onto the pull path, which is why no imagePullSecret appears in this design.

The Pod Security Admission lesson from 07-26 supplies the pattern being reused. That lesson gated on how the container asks to run. This one gates on what the container contains. Same chain, different predicate.

§VIConnection to Today's Dev Lesson

The Gatekeeper constraint above enforces a prefix and the presence of a digest. It cannot resolve a tag to a digest, because a policy engine evaluating an admission request has no business making an outbound call to a registry mid-request.

Today's Dev lesson builds the piece that does. It walks the registry HTTP API in Python: authenticate, request a manifest by tag, read the Docker-Content-Digest header, and pin. Then it wires that resolution into a validating webhook that rejects a tag-referenced image with a message naming the digest the author should have written. The Ops side draws the boundary. The Python side does the resolution the boundary depends on.

Today's Cert lesson takes the same ground through the CKS Supply Chain Security domain, where the exam asks for trivy output files, SBOM generation with bom and trivy sbom, and a correctly wired ImagePolicyWebhook with implicit-deny set.

§VIIClosing

The cluster's controls have all been about behavior. Who calls, what the call may touch, which packets cross, how the process runs. Behavior is the visible half. The image is the half that arrived before anyone was watching, and every behavioral control in the cluster is enforcing rules on code it never read.

Two moves close the gap, and neither is exotic. Deploy digests, so the artifact that was examined is the artifact that runs. Enforce the registry boundary at admission, so the examination cannot be skipped by anyone with push rights and a hurry.

Then re-scan what is deployed on a schedule, because the report from build day is a claim about a database that has since moved.

Examine the manifest of the next thing you ship. If it names a tag, you do not know what is running.