CKA Troubleshooting — the four floors
The exam does not ask you to define a CrashLoopBackOff. It hands you a broken cluster and fifteen minutes.
§I — Frame
Thirty percent. No other CKA domain is larger, and no other domain is scored so much on procedure and so little on recall.
The exam does not ask you to define a CrashLoopBackOff. It hands you a broken cluster and fifteen minutes. Candidates who have memorised YAML and never broken anything lose here, because the skill being tested is the order in which you look.
So the order is the lesson.
Ask one question before touching anything: which floor is the failure on? There are four, and they are diagnosed with four different tools.
| Floor | What lives here | The tool |
|---|---|---|
| Control plane | apiserver, scheduler, controller-manager, etcd | crictl, journalctl, the manifest files |
| Node | kubelet, container runtime, sysctls, disk | systemctl, journalctl -u kubelet |
| Workload | Pods, containers, probes, images, volumes | kubectl describe, kubectl logs |
| Network | Services, endpoints, DNS, policy | kubectl get endpoints, exec plus curl |
Call them the four floors. Naming the floor first is worth more than any single command below it, because every floor has its own log and looking in the wrong one costs the exam clock.
§II — The Control-Plane Floor: no controller is coming
The single fact that makes control-plane troubleshooting different: the control plane has no controller.
kube-apiserver, kube-scheduler, kube-controller-manager, and etcd on a kubeadm cluster are static Pods. The kubelet reads their manifests from a directory on disk, usually /etc/kubernetes/manifests, and runs them directly. No Deployment. No ReplicaSet. Nothing in the API is watching them (Poulton, Ch. 4, Pod theory, p. 45).
Three consequences follow, and every one of them appears on the exam.
**You cannot fix a static Pod with kubectl edit.** The mirror Pod visible in kube-system is a read-only reflection. Edit the file on disk.
You do not restart it. Save the manifest and the kubelet notices within seconds and recreates the Pod. Nothing to apply, nothing to rollout.
**When the apiserver is down, kubectl is down.** This is the case candidates panic in. The cluster is still there; the door is locked. Diagnosis moves to the node's own tools:
sudo crictl ps -a | grep apiserver
sudo crictl logs <container-id>
sudo journalctl -u kubelet -n 80 --no-pager
sudo cat /etc/kubernetes/manifests/kube-apiserver.yamlThe Bootcamp's Question 15 is exactly this scenario, and it is worth walking. A cluster is migrated. The apiserver will not come up. The cause is a flag: --etcd-servers points at 2380, the etcd peer port, rather than 2379, the client port. Peer traffic is etcd talking to etcd. Client traffic is everyone else. One digit, whole control plane down.
The repair sequence: read the apiserver logs, find the connection refusal, open the manifest, correct the port, save, wait for the kubelet to recreate the Pod. Then check the scheduler, because a migration that broke one component's flags usually broke another's kubeconfig path or certificate reference.
The lesson underneath the digit: a static Pod that will not start is almost always a wrong flag or a missing file, not a bug. Read the manifest against what the logs say the process could not reach.
§III — The Node Floor: NotReady has a small number of causes
A node reports NotReady when the kubelet stops posting its status. The kubelet posts status when it is running and healthy, so the question collapses to why the kubelet is unhappy.
kubectl get nodes
kubectl describe node worker-2 | sed -n '/Conditions/,/Addresses/p'The condition table is the answer sheet, and the exam expects fluency in it:
| Condition | True means |
|---|---|
Ready | kubelet healthy and accepting Pods |
MemoryPressure | node is low on memory |
DiskPressure | node is low on disk or inodes |
PIDPressure | too many processes |
NetworkUnavailable | node network is not correctly configured |
If Ready is False or Unknown, go to the node itself:
systemctl status kubelet
journalctl -u kubelet -n 100 --no-pager
systemctl status containerd
crictl infoFour common causes, counted, in the order they actually occur:
- The kubelet service is stopped or crash-looping.
systemctl start kubelet, then read the log to learn why it stopped. - The container runtime is down. The kubelet is healthy and has nothing to talk to.
crictl inforeturns an error, and that error is the real one. - Certificates expired. kubeadm client certs last one year. The kubelet log says the connection to the apiserver was refused for authentication reasons, which reads like a network problem and is not.
- The disk is full.
DiskPressureisTrue, the kubelet begins evicting Pods, and image pulls fail.
The Bootcamp's Question 9 is the runtime case built from the ground up: install the runtime shim package, enable the service, then set the sysctls that Kubernetes networking depends on. Those sysctls are worth memorising because a node that comes up without them looks healthy and drops traffic:
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
net.netfilter.nf_conntrack_max = 131072bridge-nf-call-iptables is the one that generates support tickets. Without it, bridged traffic bypasses iptables entirely, which means Service routing and NetworkPolicy both silently fail to apply. The 08-01 lesson built default-deny NetworkPolicy on the assumption that the enforcement point works. This sysctl is that assumption.
§IV — The Workload Floor: the three-rung ladder
Most exam troubleshooting tasks live here, and the procedure is short enough to run without thinking.
**Rung one: describe.** Events at the bottom, newest last. This is where scheduling failures, image pull failures, probe failures, and mount failures all announce themselves.
**Rung two: logs.** The application's own account of what it did before it stopped.
**Rung three: logs --previous.** For a container that has already restarted, the current log is from the new attempt and is usually empty or short. The evidence is in the dead container.
kubectl describe pod api-7d9f
kubectl logs api-7d9f -c app
kubectl logs api-7d9f -c app --previousThat third command is the single highest-value line in the domain. CrashLoopBackOff means the container has already died more than once, so by definition the interesting log belongs to a container that no longer exists.
A short reason-to-cause table worth holding whole:
| Status or reason | Where it actually broke |
|---|---|
ImagePullBackOff / ErrImagePull | Registry auth, wrong tag, or no network from the node |
CrashLoopBackOff | The process exits. Read logs --previous |
CreateContainerConfigError | A referenced ConfigMap or Secret key does not exist |
Pending + FailedScheduling | Scheduler: resources, taints, affinity, volume zone conflict |
Pending + FailedAttachVolume | Attach: volume busy on another node, or CSI failure |
Running but 0/1 ready | Readiness probe failing. The app started and is not answering |
The last row separates candidates. Running 0/1 is not a crash. The container is alive and the readiness probe is returning something other than success, so the Service removed the Pod from its endpoints and traffic stopped. Read the probe definition, then run the probe's own request from inside the Pod.
Today's Ops lesson takes the two Pending rows apart in depth on EKS, and adds the case this table cannot show: for a provisioning failure the Pod's own events say only that it is waiting, while the explanation sits on the PersistentVolumeClaim. When a Pod waits on storage, describe the claim.
§V — The Network Floor: endpoints before DNS
Service problems have one first command, and it is not a DNS query.
kubectl get endpoints my-svcEmpty endpoints means the Service selector matches no ready Pod. Either the labels disagree, or the Pods are Running 0/1 and readiness has removed them. Both are workload-floor problems wearing a network-floor costume, and checking endpoints first saves the ten minutes otherwise spent debugging DNS that was never broken.
If endpoints are populated and traffic still fails, work outward: exec into a client Pod, curl the Pod IP directly, then the Service IP, then the Service DNS name. Each step that succeeds eliminates a layer. The Bootcamp's Question 17 uses this shape as its verification, checking that TLS 1.2 fails and TLS 1.3 succeeds against the same host, which is the general technique of proving a fix by demonstrating both the working and the broken path rather than only the working one.
The 08-04 lesson covered RBAC and is worth recalling as a false-positive source here: a Forbidden error looks like a broken service and is an authorization decision working correctly.
§VI — Practice Questions
kubectl returns a connection refused. You have SSH access to the control-plane node. Name the first three commands you run and say what each rules out.sudo crictl ps -a | grep apiserver shows whether the container exists and whether it is exiting. sudo crictl logs <id> gives the process's own error, which usually names the resource it could not reach. sudo journalctl -u kubelet -n 80 shows whether the kubelet is even trying, which distinguishes a bad manifest from a crashing process. Only then open /etc/kubernetes/manifests/kube-apiserver.yaml and compare the flags against what the logs said failed./etc/kubernetes/manifests/kube-scheduler.yaml to fix a bad kubeconfig path. What do you run to restart the scheduler Pod?kubectl verb that applies. If the Pod does not come back, the manifest is invalid YAML or the kubelet is not running; check journalctl -u kubelet.Ready=Unknown. systemctl status kubelet reports active and running. Where do you look next and why?crictl info; an error there is the actual fault. The kubelet log will corroborate with repeated runtime-connection failures.CrashLoopBackOff. kubectl logs returns nothing. Explain why and give the command that produces evidence.kubectl logs <pod> -c <container> --previous reads the terminated instance's log, which is where the exit cause is.curl to a Service name fails from another Pod. Endpoints for the Service are empty, and kubectl get pods shows all three replicas Running with 0/1 ready. What is broken, and what is not broken?net.bridge.bridge-nf-call-iptables = 1. Without it, bridged traffic bypasses iptables, so Service routing and NetworkPolicy never apply. The node looks healthy and drops traffic. Set it alongside net.ipv4.ip_forward = 1, and persist both in /etc/sysctl.d/ so a reboot does not undo the fix.§VII — Closing
Thirty percent of the exam rewards a habit rather than a fact.
Name the floor. Control plane, node, workload, or network. Then use that floor's tool and no other. On the workload floor run describe, logs, logs --previous, in that order, every time. On the network floor check endpoints before DNS. On the control-plane floor remember that nothing is watching the static Pods, so the file on disk is both the diagnosis and the cure.
Break a cluster on purpose this week. Change the --etcd-servers port to 2380, then find your way back without reading these notes. The exam scores the return trip.
Related
- Prior arc: CKS Supply Chain Security
- Domain hub: Cross-References/domains/01-Earth-DevOps
- Grounding reference: CKA Q15 — Etcd Fix (static-pod manifest repair)
🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura Filed 2026-08-10 at Fajr. Trio #85, sprint day 19, K8s track, k8s_day_counter 6 (CKA-emphasis).