Resilient Python Ops Clients — the client that knows when to stop

A client that retries hard is indistinguishable, from the server's side, from an attack.

The physician who will not stop treating is the one the patient does not survive.

A client that retries hard is indistinguishable, from the server's side, from an attack.

§IFrame

Three rungs of this arc built a tool that behaves.

The July visit gave it a shape: a package with an entry point, importable by the rest of the fleet rather than copied between machines. The Sunday visit gave it a voice: JSON lines on stderr, a heartbeat file, an exit code that means something to the process that called it. Wednesday gave it a mouth to feed itself with: layered configuration precedence, a validation gate that exits 2 before any work begins, one paginated sweep of Parameter Store so every secret is resolved up front.

Read those three back and notice what each assumes. The package assumes the call inside it completes. The log line assumes there was a result to log. The startup validation assumes that after validation, the API answers.

Today the API does not answer.

A 503 comes back from the Cloud Storage JSON API. Then another. The tool has a try block around it, because every ops tool has a try block around it, and inside that block sits the move every engineer makes without thinking: sleep a second, try again, three times, then give up.

That reflex is the subject. It is correct in outline and wrong in all four of its details, and the failure mode it produces has a name in the SRE book that every operator should be able to recite. Call the thing being built here the client that knows when to stop.

§IIFoundations: four facts about a failed call

Fact one. Retrying a failure you caused makes the failure worse.

The SRE book's treatment of cascading failure puts the arithmetic plainly. A server under load starts returning errors. Every client retries three times. The server now receives four times its previous request volume at exactly the moment it has least capacity to serve it. The retry is a load multiplier, and it multiplies precisely when multiplication is fatal.

State it as a rule and keep it: a retry is new load, not a second chance. The client experiences it as one logical operation. The server experiences it as N requests.

Fact two. Not every error is worth retrying, and the status code tells you which.

The distinction that matters is whether the failure is about the request or about the moment. A 503 Service Unavailable, a 429 Too Many Requests, a 500, a connection reset, a read timeout: these describe the moment. Retry them. A 400 Bad Request, a 403 Forbidden, a 404, a 409 Conflict on a non-idempotent write: these describe the request. Retrying them is a loop that burns quota and ends in the same place.

Google's APIs are explicit about this. A 429 on the Compute Engine API means a quota or rate limit was exceeded, and it is retryable after a wait. A 403 with reason quotaExceeded on a daily quota is the same status class and is not retryable today, because the quota resets tomorrow. Read the reason field, not only the status.

Fact three. Uniform backoff synchronizes clients into a wave.

This is the fact most engineers miss, and it has a shape worth seeing. Fifty instances of an ops tool fire against the same API. The API has a bad second and returns 503 to all fifty. Every client sleeps exactly one second and retries. All fifty arrive again at the same millisecond. The API has another bad second. They sleep two seconds. All fifty arrive together again.

The clients have accidentally organized themselves into a synchronized attack, and they will stay synchronized until something breaks the symmetry. That something is jitter: randomness added to each sleep so the wave disperses into a spread. The SRE book's retry guidance treats randomized backoff as mandatory rather than as a refinement. Coin it: jitter is what turns a wave back into rain.

Fact four. A per-call retry limit does not bound total retry load.

Three retries per call sounds bounded. Run ten thousand calls and it is thirty thousand retries, all of them landing on a server that is already failing. The SRE book's answer is a retry budget: a server-wide or client-wide cap expressed as a ratio, commonly around ten percent of the base request rate. When retries exceed the budget, the client stops retrying entirely and fails fast, even on retryable errors.

The budget is the difference between a client that is polite per-call and a fleet that is polite in aggregate.

§IIIMechanism: the four controls and where each attaches

Four controls, each attaching at a different layer, each answering a question the others cannot.

The timeout attaches to the single call. It answers: how long am I willing to wait for one attempt? A client without a timeout is a client that hangs forever on a half-open socket, holding a worker thread and a file descriptor while the operator sees nothing at all. Set two: a connect timeout in the low single-digit seconds and a read timeout sized to the operation's real p99. Set them explicitly, because the library default is usually None.

The retry policy attaches to the logical operation. It answers: which failures do I re-attempt, how many times, and how long do I wait between them? Exponential backoff with full jitter is the standard shape. The base delay doubles each attempt, and the actual sleep is drawn uniformly from zero to that ceiling.

The retry budget attaches to the process. It answers: across all operations this process performs, how much of my traffic is allowed to be retry traffic? A token bucket, refilled at a fraction of the success rate, drained by each retry. Empty bucket means no retries until successes refill it.

The circuit breaker attaches to the dependency. It answers: is this remote service worth calling at all right now?

Newman's resiliency chapter gives the breaker its three states, and the third is the one that does the real work. Closed: calls flow through, failures are counted. Open: the failure threshold was crossed, and calls fail immediately without touching the network. Half-open: after a cooldown, exactly one probe call is permitted. If it succeeds the breaker closes; if it fails the breaker re-opens and the cooldown restarts.

The half-open state is what separates a breaker from a mute button. Without it, the breaker either stays open forever or reopens the floodgates all at once the instant the timer expires.

There is an ordering to these four, and getting it backwards produces a tool that looks correct and behaves badly. The breaker wraps the retry loop. The retry loop wraps the timed call. Put the breaker inside the retry loop and each retry consults a breaker that a prior retry just tripped, which reads as a bug and debugs as a nightmare.

§IVWorked Example: a Cloud Storage client that stops

Take the config-and-secrets tool from Wednesday's lesson and give it a remote dependency: it reads an object from a Cloud Storage bucket on every run.

Start with the classification, because everything else depends on it. Retryable statuses for the Cloud Storage JSON API are 408, 429, 500, 502, 503, 504, plus the transport-level errors that never produce a status at all.

import random, time
from dataclasses import dataclass

RETRYABLE_STATUS = frozenset({408, 429, 500, 502, 503, 504})

@dataclass(frozen=True)
class RetryPolicy:
    max_attempts: int = 4
    base_delay: float = 0.5
    max_delay: float = 20.0
    deadline: float = 60.0

The policy is a frozen dataclass and not four loose keyword arguments. It travels as one object into whatever calls it, which means the same numbers govern every call site rather than drifting per function. Today's Dev lesson takes that object apart and rebuilds it with descriptors, so the invalid values never survive construction.

Full jitter is one line, and the line is the whole idea.

def sleep_for(attempt: int, p: RetryPolicy) -> float:
    ceiling = min(p.max_delay, p.base_delay * (2 ** attempt))
    return random.uniform(0.0, ceiling)

The ceiling doubles. The sleep is drawn uniformly beneath it. Fifty clients at attempt 3 draw fifty different numbers from zero to four seconds, and the wave is gone.

The retry loop carries a deadline as well as an attempt count. An attempt count alone permits four attempts with twenty-second sleeps to consume a minute and a half of wall clock inside a tool the operator expected to finish in ten seconds.

def call_with_retry(fn, policy: RetryPolicy, budget, logger):
    started = time.monotonic()
    last_error = None
    for attempt in range(policy.max_attempts):
        try:
            result = fn()
            budget.record_success()
            return result
        except TransientError as err:
            last_error = err
            if attempt == policy.max_attempts - 1:
                break
            if not budget.try_consume():
                logger.warning("retry_budget_exhausted", extra={"attempt": attempt})
                break
            delay = sleep_for(attempt, policy)
            if time.monotonic() - started + delay > policy.deadline:
                logger.warning("retry_deadline_exceeded", extra={"attempt": attempt})
                break
            logger.info("retrying", extra={"attempt": attempt, "delay_s": round(delay, 3)})
            time.sleep(delay)
        except PermanentError:
            budget.record_success()
            raise
    raise RetryExhausted(last_error)

Three exits, each with a distinct log event. Budget exhausted, deadline exceeded, attempts exhausted. Sunday's structured-logging discipline pays for itself here: an operator grepping retry_budget_exhausted across the fleet learns something a bare stack trace never tells them, which is that the problem is not this tool but the aggregate.

PermanentError calls record_success() before re-raising. This looks wrong and is deliberate. The budget measures whether the dependency is drowning in retries, and a 403 is a clean, fast answer from a healthy server. Counting it against the budget would open the breaker on a permissions bug.

The breaker sits outside all of it.

class Breaker:
    def __init__(self, threshold=5, cooldown=30.0):
        self.threshold, self.cooldown = threshold, cooldown
        self.failures, self.opened_at, self.half_open = 0, None, False

    def before(self):
        if self.opened_at is None:
            return
        if time.monotonic() - self.opened_at < self.cooldown:
            raise CircuitOpen("gcs")
        self.half_open = True

    def after_success(self):
        self.failures, self.opened_at, self.half_open = 0, None, False

    def after_failure(self):
        if self.half_open:
            self.opened_at, self.half_open = time.monotonic(), False
            return
        self.failures += 1
        if self.failures >= self.threshold:
            self.opened_at = time.monotonic()

before() raising CircuitOpen is the point. The open breaker costs no socket, no DNS lookup, no thread parked on a read. It fails in microseconds, and Wednesday's exit-code discipline turns that into a specific non-zero code the calling process can branch on. A cron wrapper that sees "circuit open" knows to skip this run rather than page anyone.

One caution the SRE book's best-practices appendix makes explicit. Retries at multiple layers of a stack multiply. If the Google client library already retries three times, and this loop retries four, one logical operation becomes twelve requests. Before wiring a retry loop around a vendor SDK, disable the SDK's own retries or set this policy to one attempt. The google-cloud-storage client accepts a retry= argument for exactly this reason.

§VConnection to Prior Lessons

Wednesday's configuration lesson argued that a tool should fail at startup rather than midway, and that a validation gate belongs before any work. The RetryPolicy object extends that argument to the failure path: the numbers governing degradation are configuration, not literals scattered through a call site, and they get validated on the same gate as everything else.

Sunday's observability lesson gave the tool structured logging, a heartbeat file, and exit codes with meaning. Every branch in today's retry loop emits a distinct event, and the breaker's open state produces a distinct exit code. The two lessons compose: the reliability controls are only useful to the extent that an operator can see which one fired.

July's packaging lesson matters more here than it looks. Retry policy is exactly the kind of thing that must live in one shared library rather than being re-implemented per tool, because a fleet where each tool invented its own backoff is a fleet with no aggregate retry behavior at all.

§VIConnection to Today's Dev Lesson

RetryPolicy above is a frozen dataclass, which stops mutation and stops nothing else. max_attempts=-1 constructs. base_delay="0.5" constructs and then explodes four attempts later inside an arithmetic expression, far from the mistake.

Today's Dev lesson builds the mechanism that closes that gap. It walks Python's attribute protocol: descriptors as the objects that intercept get and set on a class attribute, __set_name__ as the hook that lets a descriptor learn its own name at class-creation time, and the difference between __getattr__ (called only when normal lookup fails) and __getattribute__ (called on every access). It ends by rebuilding the policy object so a negative attempt count raises at the assignment that caused it.

The Ops side sets the numbers. The Python side makes the numbers impossible to get wrong.

§VIIClosing

The reflex at the top of this lesson was: sleep a second, try again, three times, give up. Compare it to what the arc now holds. Classify the error before deciding. Draw the sleep from a jittered ceiling rather than a constant. Bound total retry load with a budget, not only per-call attempts. Stop calling the dependency entirely when it is clearly down, and probe once before believing it recovered.

Newman's chapter carries a line worth taking literally: the worst failure is the one where a client keeps a dying service dying. Every control here exists to make the client the first thing that gives up rather than the last.

Open the ops tool you run most often and find its outbound call. Check whether it sets a read timeout. Most do not, and that is the cheapest of the four controls to add.