Python and the Kubernetes Event Stream — the rumour with a timestamp

A tool that reports nothing looks exactly like a cluster with nothing wrong.

§I — Frame

The 07-29 lesson built a watch client that worked. It ran for about an hour.

Then the API server dropped its cached history, the stream ended, the for loop ran off the end of the iterator, and the process exited zero. No traceback. No alert. A tool that reports nothing looks exactly like a cluster with nothing wrong.

That failure has a name in the API: 410 Gone. It is not an error condition to be handled defensively. It is the documented, expected, routine end of every long watch, and a client that does not plan for it is a client that has not been written yet.

Today the tool gets its second half, and gets pointed at a harder stream.

§II — Language Idiom: the watch is a generator that ends

Python's client library is generated from the Kubernetes OpenAPI specification (Hightower et al., Ch. 13, OpenAPI and generated client libraries, p. 216). That generation is why the method names read like the REST paths and why the returned objects carry the API's field names verbatim. It also means the library hands you a watch.Watch().stream(...) that behaves as an iterator, and Python programmers already know exactly what an iterator that stops means.

Here is the idiom worth holding: an unbounded stream that terminates is two objects, not one. The inner object is finite and knows how to end. The outer object is infinite and knows how to start the inner one again with the right continuation token. Python spells the outer object as a generator function that yields from the inner one in a loop.

from kubernetes import client, config, watch
from kubernetes.client.exceptions import ApiException

def resilient_stream(list_fn, **kwargs):
    resource_version = ""
    while True:
        w = watch.Watch()
        try:
            for event in w.stream(list_fn, resource_version=resource_version,
                                  timeout_seconds=300, **kwargs):
                resource_version = event["object"].metadata.resource_version
                yield event
        except ApiException as exc:
            if exc.status == 410:
                resource_version = ""
            else:
                raise
        finally:
            w.stop()

Three things in that block carry weight. The resource_version is captured from each event as it passes, so a restart resumes at the last object actually seen rather than replaying the world. Setting it back to "" on a 410 is a deliberate full resync: the history the server would need is gone, so the honest move is to ask for current state again and accept the duplicate ADDED events that follow. And w.stop() in a finally closes the underlying HTTP connection, which matters because the library holds a socket per watch and a leaked one survives the loop iteration that created it.

The caller never sees any of this. The caller writes for event in resilient_stream(v1.list_pod_for_all_namespaces): and gets a stream that does not end. That is the generator earning its keep: the restart discipline lives in one place, and every consumer inherits it.

§III — Code Worked Example: the Pending triage tool

The Ops lesson establishes that Pending covers three failures owned by three controllers, and that the evidence for the second one sits on the PersistentVolumeClaim rather than the Pod. A tool that reads only Pod events will therefore be confidently wrong about the most common storage failure in the cluster.

So the tool watches two streams and correlates them.

Start with the classification, which is the part worth getting right before any I/O exists:

from dataclasses import dataclass, field

@dataclass
class Verdict:
    pod: str
    namespace: str
    owner: str
    reason: str
    detail: str
    claims: list = field(default_factory=list)

SCHEDULER_REASONS = {"FailedScheduling", "Unschedulable"}
ATTACHER_REASONS  = {"FailedAttachVolume", "FailedMount", "VolumeBindingFailed"}
PROVISIONER_REASONS = {"ProvisioningFailed", "ExternalProvisioning"}

The three sets are the Ops lesson's taxonomy rendered as data. Keeping them as module-level sets rather than a chain of if reason == comparisons means a new reason string is a one-line edit, and the sets can be asserted against in tests without standing up a cluster.

Pod conditions are the second input. A Pending Pod carries a PodScheduled condition, and its status field is the cluster's own answer to whether the scheduler is the blocker:

def scheduled_condition(pod):
    for cond in (pod.status.conditions or []):
        if cond.type == "PodScheduled":
            return cond
    return None

Returning the condition object rather than a boolean keeps reason and message available to the caller. A boolean would throw away the two fields that make the verdict useful.

Now the correlation. Claims a Pod depends on are readable straight off the Pod spec, so the tool never has to guess which PVC to blame:

def claims_for(pod):
    names = []
    for vol in (pod.spec.volumes or []):
        pvc = getattr(vol, "persistent_volume_claim", None)
        if pvc is not None:
            names.append(pvc.claim_name)
    return names

The classifier takes a Pod, the events already seen for that Pod, and the events already seen for its claims, then answers the only question the operator cares about:

def classify(pod, pod_events, claim_events):
    ns, name = pod.metadata.namespace, pod.metadata.name
    claims = claims_for(pod)

    for ev in reversed(claim_events):
        if ev.reason in PROVISIONER_REASONS:
            return Verdict(name, ns, "csi-provisioner", ev.reason, ev.message, claims)

    for ev in reversed(pod_events):
        if ev.reason in ATTACHER_REASONS:
            return Verdict(name, ns, "attachdetach-controller", ev.reason, ev.message, claims)
        if ev.reason in SCHEDULER_REASONS:
            return Verdict(name, ns, "kube-scheduler", ev.reason, ev.message, claims)

    cond = scheduled_condition(pod)
    if cond is not None and cond.status == "False":
        return Verdict(name, ns, "kube-scheduler", cond.reason or "Unschedulable",
                       cond.message or "", claims)
    return Verdict(name, ns, "unknown", "NoSignal",
                   "Pod is Pending with no explanatory event yet.", claims)

The ordering is the argument. Claim events are checked first because a provisioning failure is the case whose evidence does not appear on the Pod at all, so checking the Pod first would let a FailedScheduling warning shadow the real cause. Within each stream the scan runs newest-first, because a Pod that was unschedulable for four minutes and is now waiting on an attach should be reported as waiting on the attach.

The event streams themselves reuse the generator from §II, with the two subscriptions kept in separate indexes keyed by the object reference every Event carries:

from collections import defaultdict

pod_events = defaultdict(list)
claim_events = defaultdict(list)

def ingest(event):
    obj = event["object"]
    ref = obj.involved_object
    key = (ref.namespace, ref.name)
    if ref.kind == "Pod":
        pod_events[key].append(obj)
    elif ref.kind == "PersistentVolumeClaim":
        claim_events[key].append(obj)

involved_object is the join key, and it is the reason this correlation is cheap. Kubernetes Events are not free-text log lines; every one of them points at the object it concerns, so grouping by (namespace, name, kind) reconstructs per-object history without parsing a single message string. Messages are for humans and for the final report. The routing runs on structure.

One honesty note the tool must carry. Events expire, one hour by default on most clusters, and the API server deduplicates repeats into a single object with an incremented count. An Event is a report that something happened recently, not a durable record that it happened. Call it the rumour with a timestamp. A triage tool built on Events is a first responder; it is not an audit trail, and it should never be described as one.

§IV — Connection to Today's Ops Lesson

The Ops lesson works one EKS cluster by hand: describe the Pod, read the tail, notice the claim is Pending, describe the claim, find the UnauthorizedOperation on the EC2 call, conclude that IRSA is missing a permission. Four commands and a piece of knowledge about which object holds the evidence.

This tool encodes the piece of knowledge. PROVISIONER_REASONS checked before ATTACHER_REASONS before SCHEDULER_REASONS is the same ordering the operator's eye performs, written down where it stops depending on whoever is on call remembering it at 3am.

The eager disk from the Ops lesson surfaces here as a kube-scheduler verdict carrying the message volume node affinity conflict. The tool does not need a special case for it. The scheduler already names the condition precisely, and the tool's job is to route the naming to the right owner rather than to re-derive it.

§V — Prior-Lesson Reach

The 07-29 lesson opened the watch API and built the client-side registry over EndpointSlices. This lesson supplies the half that lesson deferred, and the fix is not specific to Events: drop resilient_stream under the 07-29 registry and it survives the hour mark too.

The 08-04 lesson established in-cluster configuration and the projected-token refresh, which is what makes config.load_incluster_config() safe to call once at startup in a long-lived process. A tool designed to run for days depends on that refresh behaviour directly.

The 08-07 lesson built a validating webhook that had to answer inside the API server's timeout. The contrast is worth naming: a webhook is synchronous and blocking, and a watch consumer is asynchronous and lagging. The webhook must be fast or the cluster stalls. The triage tool may be slow and merely reports late.

§VI — Closing

Two rules survive this lesson if nothing else does.

A watch that does not handle 410 is a watch that silently stops. Wrap it in a generator that owns the continuation token, and every consumer written afterwards inherits the fix.

Evidence lives on the object that failed, not on the object you noticed. The Pod is where you look. The claim is where the answer is.

Run the tool against a namespace you believe is healthy. The interesting output is the Pod nobody had noticed was Pending.

Related

• Prior arc: Python and the Image Admission Gate
• Language hub: Cross-References/dev-languages/Python
• Grounding tome: Kubernetes: Up and Running, 3rd Edition (Ch. 13 Accessing Kubernetes from Common Programming Languages, pp. 216-231)

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura Filed 2026-08-10 at Fajr. Trio #85, sprint day 19, K8s track.