Kubernetes Identity and Authorization on GKE — the pod's two identities

The wall you built on Saturday asks where a packet came from. It never asks who sent it.

§ IFrame

Five rungs of the K8s arc are behind us and every one of them dodged the same question. The loop taught how the cluster acts. The gate taught how it judges. The Services lesson taught how it names. Saturday's NetworkPolicy lesson taught how it separates, with walls made of labels.

Put those four together and notice the hole. A packet reaching the API server passes the network wall because its source pod carries the right label. Then something has to decide whether the sender may create a Secret in trading, or only read ConfigMaps in staging, or nothing at all. Labels cannot answer that.

Poulton draws the request path as three gates in series: authentication, then authorization, then admission (ch. 13, p. 176). The 07-26 lesson covered gate three and quietly assumed gates one and two had passed. Today pays that debt. Carry out the pod's two identities: a ServiceAccount inside the cluster, an IAM principal outside it, and a bridge that replaces the exported key file readable by root in every container on the node (Rice, ch. 12, p. 166).

§ IIFoundations — Who Is Asking

Humans are not cluster objects. There is no User resource and no kubectl create user. The API server authenticates humans externally (client certificates, OIDC, static token files) and Kubernetes only ever sees the resulting username and group strings (Poulton, ch. 13, p. 177). The cluster stores nothing about a human. It reads a name off a credential and moves on.

Workloads are cluster objects. A ServiceAccount is a real namespaced resource. Every namespace gets a default at creation and every pod that names none is mounted with it (Burns et al., ch. 14, p. 247). That default is the first thing to fix. Create a purpose-built ServiceAccount per workload and turn off automatic token mounting where the workload has no API business.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ledger-writer
  namespace: trading
automountServiceAccountToken: false

Set it false on the ServiceAccount and every pod using it starts tokenless. A pod that genuinely needs API access flips it back on in the pod spec, which makes the need visible in review rather than assumed by default.

The token has also changed shape. Modern clusters mount a projected token: audience-scoped, time-bound, refreshed in place by the kubelet, invalidated when the pod dies. The old behavior created a long-lived Secret that stayed valid after the pod was gone. Read the coin off that change: the token is a lease, not a key. Today's Dev lesson builds against exactly that refresh behavior.

§ IIIRBAC — Four Objects, One Sentence

Poulton spends seven pages on RBAC (pp. 178-184) and the scheme reduces to one shape. Roles say what. Bindings say who. On the scope axis, Role and RoleBinding live in a namespace while ClusterRole and ClusterRoleBinding apply cluster-wide. On the function axis, the Role kinds enumerate operations and the Binding kinds attach them to subjects.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ledger-writer
  namespace: trading
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    resourceNames: ["ledger"]
    verbs: ["get", "patch"]

apiGroups names the API group, where "" is the core group that catches out nearly everyone the first time. resources is plural and lowercase. verbs names the operations. Add resourceNames and the grant narrows to specific objects, which is how a controller earns the right to patch its own Deployment without earning the right to patch anyone else's.

Three properties worth memorizing

RBAC is purely additive. There is no deny rule. Effective permissions are the union of every binding naming the subject. An over-broad ClusterRoleBinding cannot be walked back by adding a narrower Role. You fix over-permission by deleting the binding.

The verb list is the actual grant. get and list are distinct powers. A subject with get on Secrets must know the name it wants; a subject with list enumerates every Secret in the namespace and reads all of them in one call. Granting list where get was meant is the most common quiet over-grant in a real cluster.

The four combinations are not symmetric. Role plus RoleBinding is the everyday case. ClusterRole plus ClusterRoleBinding is the cluster-wide grant. ClusterRole plus RoleBinding is the useful third: define the permission set once, bind it into one namespace, reuse across twenty without twenty copies of the rules. The fourth does not exist, because a namespaced Role carries rules with no meaning outside the namespace holding them.

Then the practical check, which is the command to reach for before writing any policy at all:

kubectl auth can-i create secrets \
  --namespace trading \
  --as system:serviceaccount:trading:ledger-writer

yes or no, computed by the API server's own authorizer rather than by reading YAML and hoping. Note the subject format. Every ServiceAccount also joins the group system:serviceaccounts:<namespace>; bind a ClusterRole to that group and you have granted it to every workload in the namespace, present and future. Do so deliberately or not at all.

§ IVWorked Example — The Second Identity on GKE

The ledger-writer pod has its in-cluster identity settled. Then it needs to write a Parquet file to a Cloud Storage bucket, and RBAC has nothing to say. Google Cloud has never heard of system:serviceaccount:trading:ledger-writer. It authorizes IAM principals.

The old answer was a service-account key: generate JSON, store it in a Secret, mount it, point GOOGLE_APPLICATION_CREDENTIALS at the file. It works, and it hands you a credential that never expires, sits on disk inside the container, is readable by root on the node regardless of the container boundary (Rice, ch. 12, p. 166), and gets copied to a laptop the first time somebody debugs the image locally.

Workload Identity Federation removes the file. The cluster's token issuer becomes a trusted OIDC provider from Google Cloud's side, and a specific KSA binds to a specific GSA through an IAM policy. When the pod's client library asks for a token, the GKE metadata server exchanges the projected KSA token for a short-lived GSA access token. Nothing is stored. Nothing is exported.

gcloud container clusters update hedronite-prod \
  --workload-pool=hedronite-prod.svc.id.goog

gcloud iam service-accounts add-iam-policy-binding \
  [email protected] \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:hedronite-prod.svc.id.goog[trading/ledger-writer]"

gcloud projects add-iam-policy-binding hedronite-prod \
  --member "serviceAccount:[email protected]" \
  --role roles/storage.objectCreator

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ledger-writer
  namespace: trading
  annotations:
    iam.gke.io/gcp-service-account: [email protected]

Read the member string in the second command carefully, because it is the whole idea in one line. hedronite-prod.svc.id.goog[trading/ledger-writer] names the workload pool, then the namespace, then the KSA. Google Cloud is not trusting the cluster. It trusts one ServiceAccount in one namespace of one cluster, and the namespace is part of the assertion. A workload that migrates to staging loses the grant on arrival, without anyone rotating anything.

The third command grants roles/storage.objectCreator and only that. Not storage.admin. The pod writes objects; it does not delete buckets. IAM shares RBAC's additive-union property, so the narrow role is chosen at grant time or not at all.

Two failures worth naming before they happen

Forgetting --workload-pool on the cluster leaves every token exchange failing with a metadata-server error that reads like a network problem and is not. Separately, the older node-level metadata concealment and the newer per-node Workload Identity setting are different knobs: a node pool created before cluster-level enablement keeps serving the node's own identity to pods, silently handing every pod on that pool whatever the node service account holds. Verify per pool, not per cluster.

§ VThe Discipline

Name the ServiceAccount. Never ship a workload on the namespace default. Turn automountServiceAccountToken off wherever the workload has no API business, which is most workloads.

Grant verbs, not resources. Write the rule from the operation the code performs. If the code calls get by a known name, do not grant list. Check with kubectl auth can-i --as, both for the yes-cases and for the no-cases you expect to hold.

Bind, do not export. Any long-lived cloud credential inside a cluster is a key someone will eventually copy. Bind the KSA to the GSA and let the metadata server mint tokens with a lifetime measured in minutes.

§ VIConnection to Today's Dev and Cert Lessons

The Dev lesson takes §III's can-i and moves it into the client: SelfSubjectAccessReview as a preflight the controller runs on startup, plus the projected-token refresh the in-cluster config depends on. The Cert lesson drills the CKA Cluster Architecture access-control half at exam shape: the four objects cold, the verification commands, kubeconfig contexts, and the CSR flow that mints a human credential. One question, three altitudes: who is asking, how the client proves it, how the exam scores it.

§ VIIClosing

A NetworkPolicy answers may this packet arrive. RBAC answers may this caller act. Both are allow-lists, both are additive, and both fail the same way, which is by being written wider than the thing they protect. The wall and the grant are one discipline wearing two shapes.

Examine well. Then run can-i against your own production namespace as the default ServiceAccount, and read what comes back.