Python Ops — Subprocess and SIGTERM the Child That Never Saw the Log Context

The hop to a child process keeps the environment. It drops the story.

The hop to a child process keeps the environment. It drops the story.

§I — Frame

Tuesday's lesson bounded four hundred coroutines inside one process. The semaphore, the connector pool, and the throttle were three ceilings on one event loop, and the lowest one won. The run_id from the 08-02 lesson rode every line because a logging.Filter sat on the handler the process itself installed.

Today the fan-out crosses a wall that asyncio never built.

The tool is a cross-account inventory. A parent process assumes a role in each member account through STS, then asks EC2 what is running. Written in boto3 it is one session and one logger. Written the way most fleets actually grow, it is a parent that orchestrates and a child that speaks the AWS CLI, because the CLI already knows pagination, --query, and the credential chain, and nobody wanted to reimplement them.

proc = subprocess.Popen(
    ["aws", "ec2", "describe-instances",
     "--region", region, "--output", "json"],
    env=child_env,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
)

The parent has a run_id. The parent has a JSON formatter. The parent has a SIGTERM handler that writes the heartbeat and returns 1. The child has the inherited environment, including the temporary credentials STS just minted, and a logger that was never configured.

Name what the hop drops. Coin it: the child that never saw the log context.

§II — Foundations: four facts about the process hop

**Fact one. subprocess.run is a conversation. Popen is a lease.**

Python for DevOps shows subprocess as the way a Python tool talks to a binary it does not own (Gift, Behrman, Deza, Gheorghiu, ch. 3, pp. 117-118). The check=True demonstration in that section is the whole point of the call: if the child fails and the parent does not ask, the parent parses an empty string and reports success. The 08-09 plan-JSON checker already spent this. Today the same line has a second reading. run waits. Popen returns a handle. A handle is a lease on a process the parent is now responsible for, including the moment the parent itself is told to stop.

run is correct when the child is short and the parent has nothing else to do. The inventory is neither. Forty accounts, one describe each, a deadline on the whole sweep: that is a set of leases, and the parent has to know where each one sits when SIGTERM arrives.

Fact two. A signal sent to a PID is not a signal sent to a family.

A terminal Ctrl-C delivers SIGINT to the foreground process group. kill <pid> and most supervisors deliver SIGTERM to one process. Children of that process keep running. They keep holding STS sessions. They keep describing instances. They keep writing to pipes whose other end is gone.

Two placements make this worse, and both are common.

start_new_session=True (or preexec_fn=os.setsid on older interpreters) puts the child in its own session. That is the right move when the child must outlive a controlling terminal. It is the wrong move when the parent is the supervisor, because the child is now unreachable by any signal the parent did not explicitly forward.

The other placement is no handler at all. The parent dies on SIGTERM. The children become orphans, reparented to PID 1, and finish on their own clock. The heartbeat the 08-02 lesson taught the parent to write on the way out now records a finished run while describes are still in flight in twelve accounts.

Fact three. The environment crosses the hop. The log context does not.

A child inherits os.environ as it stood at Popen time, plus whatever env= the parent passed. Temporary credentials, AWS_REGION, HTTPS_PROXY: those survive. The in-memory logging tree does not. Handlers, formatters, filters, LoggerAdapter extras, the run_id the 08-02 lesson attached with a filter: all of that lives in the parent's address space. The child starts with the last-resort handler, which writes a different shape to a different stream, with no run_id and no account id.

If the parent captures stdout and logs it, the child's payload can be wrapped. If the child writes its own logs, or if the parent dumps stderr through unfiltered, the stream that Tuesday's governor could key on becomes two dialects in one file. Grep for the run_id and the child's lines vanish. That is the silent success wearing a new coat: the run happened, the record is incomplete, and the incomplete record looks like a run that never touched those accounts.

Fact four. The child's exit code is not the parent's witness.

08-02 made the exit code a computed verdict at the entry point. A parent that ignores proc.returncode and returns 0 has lied to cron. A parent that writes the heartbeat before proc.wait() has dated a completion that has not occurred. A parent that calls communicate() with no timeout can hang past the supervisor's own deadline, so the supervisor SIGKILLs the parent and the children from fact two keep going.

The three witnesses still hold. Each of them now has a child-shaped failure. The log line without a run_id. The heartbeat written while a lease is open. The exit code that never asked the child.

§III — Mechanism: where the signal goes, and what the child is told

The parent owns three jobs the moment Popen returns: feed or close the child's stdin, drain stdout and stderr so the pipe cannot fill and stall, and decide what SIGTERM means for the lease.

A handler that only sets a flag is unfinished. The flag has to reach the children.

class ChildLease:
    def __init__(self, args, env, log):
        self.log = log
        self.proc = subprocess.Popen(
            args,
            env=env,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            start_new_session=True,
        )

    def wait(self, timeout):
        try:
            out, err = self.proc.communicate(timeout=timeout)
        except subprocess.TimeoutExpired:
            self.terminate()
            out, err = self.proc.communicate(timeout=5)
        return self.proc.returncode, out, err

    def terminate(self):
        if self.proc.poll() is not None:
            return
        try:
            os.killpg(self.proc.pid, signal.SIGTERM)
        except ProcessLookupError:
            return
        try:
            self.proc.wait(timeout=8)
        except subprocess.TimeoutExpired:
            os.killpg(self.proc.pid, signal.SIGKILL)
            self.proc.wait(timeout=2)

Three decisions sit in that block, and each one is a sentence you write down so the next reader does not have to rediscover it.

start_new_session=True is used because the parent will forward. The child is in its own group so a SIGTERM aimed at the child cannot bounce back into the parent. The parent then becomes the only thing allowed to signal that group, which is the point of the lease.

communicate(timeout=...) is the drain and the wait in one call. A wait() without a drain leaves the child blocked on a full pipe. A drain without a timeout leaves the parent blocked on a child that will not finish.

killpg then SIGKILL is the two-step stop. SIGTERM is the request. Eight seconds is the courtesy. SIGKILL is the fact. The 08-08 lesson taught a retry budget for a sick server; this is the same budget pointed at a sick child. Retrying a child the parent already SIGTERM'd is how a sweep doubles its STS traffic on the way down.

The log context is the other half of the hop, and it is a wrap, not a hope:

def emit_child(log, account, returncode, out, err):
    log.info(
        "child finished",
        extra={
            "account": account,
            "child_rc": returncode,
            "stdout_bytes": len(out or b""),
            "stderr_head": (err or b"")[:400].decode("utf-8", "replace"),
        },
    )

The child is not asked to log. The parent logs about the child, on the parent's handler, with the parent's run_id still attached by the 08-02 filter. The payload can be parsed in the next section. The story stays in one dialect.

§IV — Worked example: the cross-account describe

The parent walks a list of account ids. For each one it assumes a role, builds a child environment that carries only the session the child needs, and leases one aws ec2 describe-instances.

def session_env(creds, region):
    env = os.environ.copy()
    env["AWS_ACCESS_KEY_ID"] = creds["AccessKeyId"]
    env["AWS_SECRET_ACCESS_KEY"] = creds["SecretAccessKey"]
    env["AWS_SESSION_TOKEN"] = creds["SessionToken"]
    env["AWS_DEFAULT_REGION"] = region
    env.pop("AWS_PROFILE", None)
    return env


def describe_account(account, role_arn, region, log, deadline):
    creds = sts.assume_role(
        RoleArn=role_arn,
        RoleSessionName=f"inv-{account}",
        DurationSeconds=900,
    )["Credentials"]
    remaining = max(5.0, deadline - time.monotonic())
    lease = ChildLease(
        ["aws", "ec2", "describe-instances", "--output", "json"],
        session_env(creds, region),
        log,
    )
    rc, out, err = lease.wait(timeout=remaining)
    emit_child(log, account, rc, out, err)
    return account, rc, out

The environment is a grant, not a dump. AWS_PROFILE is removed so a leftover profile on the operator's laptop cannot override the assumed session. Duration is nine hundred seconds because a child that outlives its credentials produces a different failure than a child that was never started, and the parent should be able to tell them apart.

The SIGTERM handler is installed once, at entry, and it holds the live leases:

leases = []

def on_term(signum, frame):
    log.warning("sigterm", extra={"live": len(leases)})
    for lease in list(leases):
        lease.terminate()
    raise SystemExit(1)

signal.signal(signal.SIGTERM, on_term)
signal.signal(signal.SIGINT, on_term)

SystemExit(1) is the computed verdict. The try/finally around main still writes the heartbeat, and the heartbeat's outcome is fail because a SIGTERM is a stop, not a success. Write it after the children have been signaled, not before. A heartbeat that lands while twelve describes are in flight is the 08-02 silent success one hop later.

Fan-out stays bounded. Tuesday's semaphore was a coroutine ceiling. Today's ceiling is a process ceiling, and it is smaller, because each child is a real AWS CLI with its own Python interpreter and its own HTTPS pool.

sem = threading.BoundedSemaphore(6)

def one(account):
    with sem:
        return describe_account(account, role_for(account), region, log, deadline)

Six is a number you measure. It is not four hundred. The 08-11 lesson closed by asking whether the fan-out was necessary at all. The same question applies: ec2:DescribeInstances across accounts is a job for a parent using boto3 and a paginator, and the child exists in this lesson so the hop can be named. In production, prefer the call you do not spawn.

§V — Connection to prior lessons

The 08-02 lesson gave the parent three witnesses. This lesson names the way each witness fails the moment a child is introduced. Attach the run_id in the parent and wrap the child's output; do not wait for the child to grow a conscience. Write the heartbeat after the last lease is reaped. Compute the exit code from the children's return codes, not from the fact that main reached its last line.

The 08-08 lesson taught a retry budget and a breaker. A child that exits 255 because STS rejected the session is a retryable call. A child the parent just SIGTERM'd is not. Classify before you retry, or the breaker never opens and the stop doubles the load.

Tuesday's semaphore still applies, pointed at processes rather than coroutines. The connector-pool warning becomes a file-descriptor warning: each Popen with two pipes spends three fds, and a fan-out of two hundred without a ceiling is a Too many open files that no AWS error explains.

§VI — Connection to today's Dev lesson

The parent in §IV receives a payload that is a mapping: an account, a return code, a body that may be a describe result or an error document. The code that classifies that payload is almost always a pile of if "Reservations" in body and if body.get("Error"). That pile looks at keys. Today's Dev lesson takes the same payload and matches the subject, so the classification names the shape instead of hunting for a field.

The hop drops the log context. The language, used carelessly, drops the shape. Both are the same mistake at two altitudes: treating the thing you received as a bag of parts instead of a thing with a form.

§VII — Closing

A child process inherits the environment and none of the story. Signal the process group you created. Drain the pipes you opened. Log about the child on the parent's handler. Write the heartbeat after the last lease is quiet.

Open the ops tool you run most often and find every subprocess.run and every Popen. For each one, ask three questions. Where does SIGTERM go. Where does the run_id go. When is the heartbeat written. Any answer that is "the child will handle it" is the child that never saw the log context.

Related

• Prior arc: Bounded Concurrency for Python Ops Tools (2026-08-11)
• Domain hub: Cross-References/domains/01-Earth-DevOps
• Grounding tome: Python for DevOps (Ch.3 subprocess pp. 117-118; Ch.7 logging pp. 290-293)

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura Filed 2026-08-14 · Fajr anchor · sprint track Python day 23 · eighth Python visit · trio #89