48 小時實測筆記：長時間模型處理任務第 31 小時以 exit code 137 終止

I run long jobs on purpose, because short jobs lie to me in pleasant ways. A loop that finishes in ninety seconds never teaches you what happens when a queue drains unevenly for two days straight. So I keep a habit: write the boring batch, let it run, and write down what actually breaks instead of what I predicted would break.

This round the job died at hour thirty-one with exit code 137, and I spent most of a day debugging a bug that did not exist. Here is what I tried, what broke, what I would repeat, and the two scripts I now keep in the repo before any multi-day run starts.

What the job was supposed to do

The workload was simple on paper: read a JSONL file of items, send each item through a model, and append the response to an output file. Nothing clever, no distributed coordination, no cleverness to hide behind. I ran it on the free server option because I needed a box that stayed up while my laptop slept, and the free models meant a two-day loop did not require me to watch a spend meter.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That combination is genuinely useful for this kind of experiment, and it is also exactly how you end up running a job on a machine whose memory ceiling you never measured. Why would you measure it? The job worked for thirty hours.

Hour 31: a very unhelpful kind of quiet

The symptoms were almost insultingly vague. No traceback, no partial output, no model error, just a process that existed and then did not. My supervisor logged one line: worker exited with code 137.

Exit code 137 is not mysterious once you decode it, and I should have done that first. A shell reports 128 plus the signal number, so 137 is 128 + 9, which is SIGKILL. Nobody kills politely with SIGKILL, which means either the kernel did it or something above me did it, and the process had no chance to write a goodbye message.

Compare that against neighbours I have confused before:

• 137 → SIGKILL (9). Kernel OOM kill, or an external supervisor.
• 143 → SIGTERM (15). A polite shutdown that my code may have swallowed.
• 1 → my bug, and honestly the one I wanted.

The wrong turn I took, in detail

Because the job called a model, I assumed the model was the problem. Doesn't that always feel like the cheapest hypothesis? I added retry logging, dumped response payloads, and re-ran the last 200 items by hand, which cost me six hours and produced nothing.

Then I re-ran the full loop with the model call stubbed out to a constant string, and it still died in the same place. That single test is the one I should have run at hour one: replacing the network with a constant tells you whether the failure is in your control flow or outside it.

Where I should have looked first: cgroup v2

On a modern Linux box, your process usually does not have the whole machine. It has a cgroup, and the cgroup has a hard memory ceiling written down in a file. Reading that file costs one command and would have saved me a day.

Here is the preflight script I now run before any long job, and it is deliberately boring:

#!/usr/bin/env bash
set -euo pipefail

echo "== cgroup version (2 means namespaced cgroupfs at /sys/fs/cgroup) =="
stat -fc %T /sys/fs/cgroup

for f in memory.max memory.current memory.peak memory.events; do
  path="/sys/fs/cgroup/$f"
  if [ -r "$path" ]; then
    printf '%-16s ' "$f"
    tr '\n' ' ' < "$path" | sed 's/ *$//'
    echo
  else
    echo "$f not readable in this container"
  fi
done

echo "== page size =="
getconf PAGESIZE

Two details matter. memory.max is the ceiling and memory.current is where you are right now, both in bytes. memory.events contains a counter named oom_kill, and if that number increments during your run, the kernel killed you and you can stop reading your own source code.

memory.peak exists on newer cgroup v2 kernels (5.19 and later), and it is wonderful when present. It is also reset when the cgroup is recreated, so treat it as a hint about the current container, not a permanent audit trail.

The artifact: a probe that reproduces the leak in ninety seconds

I did not want to wait thirty hours to learn the same lesson again, so I wrote a small harness that compares retaining chunks in a list against streaming them to disk.

# leak_probe.py -- labelled demo harness, not production code
import resource
import sys

# ru_maxrss is KiB on Linux and bytes on macOS -- do not compare across both blindly.
def rss_kib():
    return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

CHUNK = b"x" * 4096


def retain(n):
    buf = []
    for _ in range(n):
        buf.append(CHUNK)
    return len(buf)


def to_disk(path, n):
    with open(path, "wb") as fh:
        for _ in range(n):
            fh.write(CHUNK)


if __name__ == "__main__":
    mode, n = sys.argv[1], int(sys.argv[2])
    if mode == "retain":
        print(f"retained {retain(n)} chunks")
    elif mode == "disk":
        to_disk("/tmp/probe.bin", n)
        print(f"wrote {n} chunks")
    else:
        raise SystemExit("mode must be retain|disk")
    print(f"ru_maxrss={rss_kib()} KiB")

Run both modes with the same n and watch the last line:

python3 leak_probe.py retain 200000
python3 leak_probe.py disk   200000

At roughly 4 KiB per chunk, the retained branch holds on the order of 800 MB of live objects, while the disk branch holds almost nothing in the heap. On a small box, the first command is an OOM kill waiting for a scheduling decision. My real job did the retained version, except the chunks were decoded model responses accumulated so a final sanity check could compare them all at the end.

Once you suspect heap retention, tracemalloc tells you the line that is holding the memory:

import tracemalloc
tracemalloc.start(25)
# ... run the suspicious section ...
snap = tracemalloc.take_snapshot()
for stat in snap.statistics("lineno")[:5]:
    print(stat)

The false positive that almost fooled me twice

Memory went up while the job wrote its output file, which looked like a leak and was not. Page cache is charged to your cgroup, so memory.current climbs when you write a lot of data, even though that memory is reclaimable under pressure. A quick comparison keeps you honest:

awk '/^(anon|file) /{print}' /sys/fs/cgroup/memory.stat

If file is what is growing and anon is flat, you have cache, not a leak, and the kernel will drop it before it kills you. If anon is climbing in step with your output size, you are holding objects alive and no amount of tuning will save you.

What broke, and what I would repeat

What broke was my assumption that a long-lived worker is a stable worker, plus my habit of trusting the most expensive hypothesis first. What I would repeat, without hesitation, is the stubbed-model re-run and the preflight read of memory.max.

My triage table, written down so future me stops improvising:

Limitations and who should skip this

This workflow assumes Linux with cgroup v2 and readable cgroup files, which is not guaranteed on every managed platform. If you cannot read /sys/fs/cgroup, fall back to sampling ps -o rss= -p <pid> and accept that you lose the OOM counter. If your job is a short request-response function, none of this pays for itself, and a simple timeout plus alert will serve you better.

Also skip the two-day loop if your job is not idempotent, because you will restart it and you will double-write something. My output writer appends with a per-item key for exactly that reason, and the check is cheaper than the cleanup.

If you want to try the free server route for a run like this, start with the preflight script before your first item is processed, not after hour thirty-one. Long jobs do not need more cleverness, they need one honest measurement taken before you go to sleep.