Kubernetes Runtime Isolation on GKE — the guest kernel

A container is a process the kernel has agreed to pretend is a machine. gVisor inserts a second kernel in user space.

§I — Frame

A container is a process the kernel has agreed to pretend is a machine. Namespaces hide the view. Cgroups cap the budget. The process still talks to the same kernel every other process on that node talks to.

That is the fact the last seven rungs of this arc have walked around. Reconciliation watches desired state. Admission judges the spec. NetworkPolicy walls the flat network. RBAC names the caller. The image scan asks what bytes were pulled. The volume path asks which disk got attached. None of those questions is "which kernel answers this syscall."

Rice puts the isolation model in one picture: from the host, a container is a tree of processes with a different root filesystem and a different network stack, and ps on the node still lists them (Ch. 4, Container Processes from the Host Perspective, pp. 50-52). The kernel is shared. A kernel bug is a node bug. A node bug is every Pod on that node.

gVisor inserts a second kernel in user space and makes the container talk to that one instead. Call it the guest kernel. Hold the name. The rest of this lesson is the map from that object to a GKE node pool, a RuntimeClass, and the syscalls that never reach Container-Optimized OS.

The exam handler is runsc. The GKE RuntimeClass name is gvisor. Those two strings are the whole contract. Everything else is placement.

§II — Foundations: a RuntimeClass names a handler, not a feeling

Burns, Beda, Hightower, and Evenson introduce RuntimeClass as the API that lets a Pod select a container runtime the way it already selects a ServiceAccount or a StorageClass (Ch. 14, RuntimeClass). The object is small on purpose:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc

Three fields, counted.

1. **kind: RuntimeClass.** Cluster-scoped. One name, cluster-wide. The kubelet on each node looks up the name and asks the CRI for the matching handler. A name with no handler on that node is a Pod that will never start.

1. **handler.** The CRI runtime name registered on the node. For gVisor the handler is runsc. The Bootcamp's Question 10 asks you to create a RuntimeClass named sandboxed with exactly this handler, then patch every Deployment in a namespace so the Pod template carries runtimeClassName: sandboxed. The name on the exam is yours to pick. The handler is not.

1. **runtimeClassName on the Pod spec.** The Pod's vote. It sits at the same level as containers:, which is why a Deployment patch has to land on spec.template.spec and not on spec. Question 10's notes flag this. Candidates who patch the Deployment spec itself watch the roll stall and blame gVisor.

Rice describes what runsc actually is (Ch. 8, gVisor, pp. 100-102). gVisor intercepts system calls the way a hypervisor intercepts a guest VM, except the intercept lives in user space. The documentation calls it a "user-space kernel." Rice flags the phrase as a contradiction in terms and then uses it anyway, because it is the accurate one: a number of Linux syscalls are implemented by gVisor itself, and the host kernel sees runsc rather than the application.

Two consequences follow from that sentence, and both show up in production.

Not every syscall exists. Rice's count at publication was 97 unimplemented syscalls. An application that needs one of them cannot run inside gVisor. This is a compatibility list, not a performance footnote. The guest kernel is a smaller kernel. Smaller is the point. Smaller is also the break.

From the host, the process tree changes shape. Rice walks ps fax against a runsc-started shell and shows runsc as the parent, with the container's sh living inside the sandbox rather than as a direct child of containerd. The Bootcamp's Question 22 makes the same proof operational: dmesg inside a sandboxed Pod prints Starting gVisor... instead of the host kernel's boot log. That one line is the isolation claim made visible.

The comparison to a virtual machine is the other foundation (Rice Ch. 5, Container Isolation Compared to VM Isolation, pp. 63). A VM has a guest kernel because the hypervisor gave it one. A container has a guest kernel only if you installed one. GKE Sandbox is Google installing one for you, with the scheduling and the taints already wired.

§III — Mechanism: Autopilot votes per Pod; Standard votes per pool

GKE Sandbox is gVisor as a managed product. The product has two shapes, and mixing them in your head is how a cluster ends up with sandboxed kube-dns.

Autopilot: the RuntimeClass is the whole move

On Autopilot you set runtimeClassName: gvisor on the Pod (or on the Pod template). GKE places that Pod on a sandboxed node. You do not create the node pool. You do not taint it. You do not write the RuntimeClass object; the cluster already has gvisor. The vote is per Pod. Two Pods in the same Deployment can disagree, and Autopilot will put them on different kinds of node.

That is the shape to reach for when the workload mix is mixed: a frontend that must be sandboxed sitting next to a sidecar that cannot be.

Standard: the pool is the sandbox

On Standard, sandbox is a node-pool property. You create a pool with --sandbox type=gvisor and --image-type=COS_CONTAINERD. Every Pod that lands on those nodes is sandboxed. The RuntimeClass still matters, because GKE taints the sandbox nodes sandbox.gke.io/runtime=gvisor:NoSchedule and the gvisor RuntimeClass carries the matching toleration and node selector. A Pod that omits runtimeClassName: gvisor will not tolerate the taint, so it will not land there. A Pod that includes it will.

Three constraints the product will not negotiate.

You cannot sandbox the default pool. The default pool runs kube-system, and kube-system includes components that need syscalls gVisor does not implement, plus hostPath mounts the sandbox refuses. Attempting to convert the default pool is a refused API call, not a warning.

You need at least one non-sandbox pool. The cluster has to put unsandboxable work somewhere. DaemonSets that touch the host, CSI node plugins, kube-proxy in some modes, and anything privileged all require a regular COS node. A Standard cluster whose only pool is a sandbox pool is a cluster that cannot finish installing itself.

**The image type is COS_CONTAINERD.** Ubuntu node images are out. The sandbox is built against Container-Optimized OS and containerd. Mix image types across pools if you must; do not mix them inside the sandbox pool.

What the sandbox refuses

The incompatibility list is the product telling you where the guest kernel stops pretending to be Linux.

• **privileged: true.** The sandbox is the opposite of privileged. The Pod will not schedule, or it will be rejected at admission.
• **hostPath.** A bind-mount of the host filesystem into the guest would punch a hole through the whole design. Refused.
• Custom seccomp and AppArmor profiles. The guest kernel has its own syscall filter. Stacking a second one from the host is undefined, so GKE refuses the combination. (The CKS exam still tests seccomp and AppArmor on unsandboxed nodes; they are a different isolation tool, Rice Ch. 8 pp. 95-99, and they do not compose with gVisor on GKE.)
• **kubectl port-forward.** The forwarding path expects a runtime the sandbox does not present. Debug some other way: logs, a temporary Service, an in-cluster curl Pod.
• **NET_RAW.** Denied. Ping from inside the sandbox fails. This surprises people who use ping as a connectivity check. Use curl against a Service instead.
• The GCE metadata server. Blocked at the OS. A sandboxed Pod that curls 169.254.169.254 does not get a token. Workload Identity is the path, which is the 08-04 lesson applied as a hard requirement rather than a preference. The metadata hop was always the weaker binding. The sandbox makes it a non-binding.

Persistent Disk CSI is the exception that looks like a contradiction and is not. The CSI node plugin runs outside the sandbox, on the host, and mounts the volume into the sandbox from there. The Pod sees a mounted filesystem. The guest kernel did not have to implement AttachVolume. Storage isolation and process isolation are two different walls, which is the 08-10 lesson standing next to this one: the disk is attached by a controller the sandbox does not contain.

§IV — Worked Example: one Standard cluster, one sandbox pool, two Pods

Cluster trading-std in us-central1. One default pool of COS_CONTAINERD nodes, already running kube-system. The trading namespace from the 08-01 NetworkPolicy lesson is about to take untrusted batch work, and that work is not allowed to share a kernel with the API that serves customers.

Step one. Create the sandbox pool.

gcloud container node-pools create sandbox-pool \
  --cluster=trading-std \
  --region=us-central1 \
  --machine-type=e2-standard-4 \
  --num-nodes=2 \
  --image-type=COS_CONTAINERD \
  --sandbox type=gvisor

GKE creates the pool, taints the nodes, and ensures the gvisor RuntimeClass exists. Confirm both:

kubectl get runtimeclass
kubectl get nodes -l cloud.google.com/gke-sandbox=gvisor

You want gvisor in the first list and two Ready nodes in the second. If the RuntimeClass is missing, stop. The pool is decorative until the name exists.

Step two. A Pod that belongs in the sandbox.

apiVersion: v1
kind: Pod
metadata:
  name: batch-untrusted
  namespace: trading
spec:
  runtimeClassName: gvisor
  serviceAccountName: batch-runner
  containers:
    - name: worker
      image: us-central1-docker.pkg.dev/prod/batch/worker@sha256:9f3a...
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        runAsNonRoot: true
        runAsUser: 65532
      resources:
        requests:
          cpu: "500m"
          memory: "512Mi"

The digest pin is the 08-07 lesson. The ServiceAccount is the 08-04 lesson: this Pod will never see the metadata server, so the only identity it can present to GCS or BigQuery is the GSA bound to batch-runner through Workload Identity Federation. The readOnlyRootFilesystem is the Cert lesson's immutability half, applied here because a sandbox that lets the process rewrite its own image is a sandbox with a writable floor.

Apply it. Confirm it landed:

kubectl get pod batch-untrusted -n trading -o wide
kubectl get pod batch-untrusted -n trading \
  -o jsonpath='{.spec.runtimeClassName}{"\n"}{.spec.nodeName}{"\n"}'

The node name should be one of the sandbox-pool nodes. Then the proof Rice and Question 22 both want:

kubectl exec -n trading batch-untrusted -- dmesg | head

You are looking for Starting gVisor. If you see the COS kernel's boot messages, the Pod is not sandboxed, and the RuntimeClass did not take. Check the pool taint and the Pod's runtimeClassName before you touch anything else.

Step three. A Pod that must not go in.

apiVersion: v1
kind: Pod
metadata:
  name: batch-privileged
  namespace: trading
spec:
  runtimeClassName: gvisor
  containers:
    - name: worker
      image: us-central1-docker.pkg.dev/prod/batch/worker@sha256:9f3a...
      securityContext:
        privileged: true

This object is a refusal. GKE will not run privileged inside the sandbox. The event list on the Pod (the 08-10 diagnostic, pointed at a different owner) will name the incompatibility. The fix is not "make the sandbox allow it." The fix is to drop privileged, or to drop runtimeClassName and accept that this Pod shares the host kernel.

The metadata check is the quieter failure, and it is the one that looks like an IAM outage:

kubectl exec -n trading batch-untrusted -- \
  wget -qO- --timeout=2 http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token \
  --header 'Metadata-Flavor: Google'

Timeout or connection refused. The OS blocked it. If this Pod was authenticating to GCS with a metadata token last week, that path is gone. Bind batch-runner to a GSA with roles/storage.objectViewer on the one bucket, the way the 08-04 lesson wired it, and the call goes through IAM without ever touching the metadata hop.

Step four. Where the CSI plugin actually runs.

The batch worker writes a result object to a PersistentVolumeClaim backed by Persistent Disk. The PVC binds (08-10: WaitForFirstConsumer, one zone). The CSI node plugin on that node, running outside runsc, attaches and mounts. The sandbox sees a filesystem. dmesg inside the Pod still says gVisor. The disk still exists on the host. Two walls, two owners, one mount path.

§V — Connection to Prior Lessons

The 08-04 GKE identity lesson built Workload Identity as the preferred binding. Today it is the only binding that still works. The metadata server was a convenience the sandbox withdraws. If a workload cannot use Workload Identity, it cannot run sandboxed, and that is a placement decision, not an IAM ticket.

The 08-07 AKS supply-chain lesson asked what bytes the kubelet pulls. The sandbox does not inspect those bytes. A sandboxed Pod running evil@sha256:abcd is a well-isolated copy of evil. Isolation and provenance are orthogonal. Run both.

The 08-10 EKS storage lesson traced four owners on the way from claim to mount. The fourth owner, the CSI node plugin, sits outside the sandbox on GKE. That is why a sandboxed Pod can still mount a Persistent Disk, and why a volume-related Pending is still diagnosed on the claim and the VolumeAttachment, not on runsc.

The 08-01 NetworkPolicy lesson still applies. NET_RAW being denied does not replace a default-deny policy. It only removes ping. The allow-list remains the wall for who can talk to whom; the guest kernel is the wall for which syscalls those talks are allowed to become.

§VI — Connection to Today's Dev Lesson

The RuntimeClass vote has to appear on every Pod that belongs in the sandbox. Typing it into each manifest works until someone ships a Deployment that forgets. The 08-01 kopf lesson installed a NetworkPolicy baseline from a CRD. Today's Dev lesson takes a narrower, meaner shape: a kopf mutating handler plus a namespace-label timer that patches runtimeClassName: gvisor onto Pods in labeled namespaces, and that names the failure mode when the patch lands after the container has already started.

Call that failure the patch that arrives too late. A mutate on create is a decision. A timer that patches a running Pod is a suggestion the kubelet may honor only by recreating the Pod, and a recreate that races a syscall is a window. The Ops lesson says where the Pod must land. The Dev lesson says who writes the field, and when the write is already useless.

Today's Cert lesson is the other half of the same wall. RuntimeClass is create-time isolation. Falco, the audit policy, and readOnlyRootFilesystem are what you still need after the process is running, because the guest kernel is a smaller kernel, not an omniscient one. The syscall that should not have happened is the Cert lesson's name for the event Falco exists to catch.

§VII — Closing

The shared kernel is the default. The guest kernel is the exception you have to ask for, by name, on the Pod, against a pool that can honor the name.

On Autopilot the ask is the RuntimeClass. On Standard the ask is the RuntimeClass plus a sandbox pool plus a non-sandbox pool you leave alone. The handler is runsc. The GKE name is gvisor. Privileged, hostPath, custom seccomp, port-forward, NET_RAW, and the metadata server are on the other side of that name, and Workload Identity is the identity path that remains.

Examine the handler. Then check which pool the Pod actually landed on, and read dmesg until it says gVisor.

Related

• Prior arc: Kubernetes Storage on EKS
• Domain hub: Cross-References/domains/01-Earth-DevOps
• Grounding tome: Container Security (Rice) (Ch. 8 Strengthening Container Isolation, gVisor, pp. 100-102)
• Paired Dev: kopf RuntimeClass enforcement
• Paired Cert: CKS Runtime Security

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura Filed 2026-08-13 at Fajr. Trio #88, sprint day 22, K8s track rung 8.