48 小時故障筆記：本機通過、遠端失敗時先比對八項環境資訊

Have you ever shipped a script that ran clean on your laptop and then failed on a remote box with nothing more useful than exit code 1? I spent a two-day window on exactly that class of failure, and the boring answer was not a bug in my code at all. The two hosts disagreed about timezone, file-descriptor budget, and default text encoding, and each disagreement broke a different part of the same job. This is the checklist I now run before I trust any second host, plus the small harness that produces it.

One honesty note before the code: the harness is the claim here, not the values. Every number you see in the sample output below is a placeholder shape, so run the capture on your own two hosts and let the diff speak for itself.

The failure shape: no traceback, just divergence

Environment drift rarely announces itself with a stack trace, which is what makes it so expensive to find. The job finishes, the log looks almost normal, and one output file is a day older than it should be. By the time you notice, you have already rewritten two functions that were never wrong.

What finally worked for me was to stop reading application logs and start comparing host facts side by side. Logs tell you what happened after the drift, but facts tell you why the same input took a different path. Once the two fact sets sit in one table, the argument about whose machine is weird ends in about thirty seconds.

Why comparing logs never converged

My first instinct was to add more logging, and that only produced more places for the two hosts to disagree. Timestamps from different offsets made the merged log unreadable, and quiet failures left no rows at all. Is it really a surprise that a log with missing rows cannot explain a missing row?

A log is an event stream filtered by whatever you thought to print, which is exactly the wrong shape for this problem. Environment facts are cheap, complete within their small scope, and identical in structure on both sides. Diffing two structured snapshots turns a narrative hunt into a table comparison.

The eight facts I capture before a remote run

I keep the list short on purpose, because a long list nobody runs is worse than a short list that always runs. These are the eight that have actually changed job behaviour in front of me:

1. Python version and binary name — a different minor version can change stdlib behaviour, and python3 may not be the interpreter your shell activation suggested.
2. Default and preferred encodings — sys.getdefaultencoding(), locale.getpreferredencoding(False), and sys.stdout.encoding are three different answers.
3. Timezone name and UTC offset — time.tzname plus the current offset catches the classic UTC-versus-local split.
4. Locale environment — LANG, LC_ALL, and LC_NUMERIC decide how numbers and dates get parsed when you do not pin them.
5. CPU count and CPU affinity — os.cpu_count() lies to you inside a restricted cgroup, so compare it with len(os.sched_getaffinity(0)).
6. File-descriptor and process limits — RLIMIT_NOFILE is the one that kills connection pools and directory walks at exactly the wrong moment.
7. Temp directory filesystem and free space — /tmp on tmpfs behaves nothing like /tmp on disk, and atomic rename across mounts is not atomic.
8. Working-directory filesystem — a network or overlay mount can change fsync cost and rename semantics under you.

The harness: one file, standard library only

This script captures those facts into JSON and then prints only the fields that differ between two snapshots. It is deliberately boring, because a tool you can read in one sitting is a tool you will actually trust.

#!/usr/bin/env python3
'''drift.py - snapshot and compare the environment facts that change job behaviour.

    python drift.py capture --out local.json
    python drift.py capture --out remote.json   # same command, other host
    python drift.py diff local.json remote.json
'''
import argparse
import json
import locale
import os
import platform
import resource
import shutil
import sys
import tempfile
import time

VOLATILE = ('time.now_utc',)


def mount_fstype(path):
    '''Best-effort fstype for path on Linux; returns unknown elsewhere.'''
    best, fstype = '', 'unknown'
    try:
        with open('/proc/mounts', encoding='utf-8', errors='replace') as fh:
            for line in fh:
                parts = line.split()
                if len(parts) < 3:
                    continue
                point = parts[1].replace('\\040', ' ')
                if path.startswith(point) and len(point) >= len(best):
                    best, fstype = point, parts[2]
    except OSError:
        pass
    return fstype


def capture():
    tmp = tempfile.gettempdir()
    free = shutil.disk_usage(tmp).free
    try:
        affinity = len(os.sched_getaffinity(0))
    except AttributeError:
        affinity = None
    return {
        'python': {
            'version': platform.python_version(),
            'implementation': platform.python_implementation(),
            'binary': os.path.basename(sys.executable),
            'hash_randomization': sys.flags.hash_randomization,
        },
        'encoding': {
            'default': sys.getdefaultencoding(),
            'preferred': locale.getpreferredencoding(False),
            'stdout': sys.stdout.encoding or 'none',
            'filesystem': sys.getfilesystemencoding(),
        },
        'time': {
            'tzname': list(time.tzname),
            'utc_offset_s': -time.timezone,
            'now_utc': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),
        },
        'env': {k: os.environ.get(k, 'unset') for k in ('LANG', 'LC_ALL', 'LC_NUMERIC', 'TZ')},
        'cpu': {'count': os.cpu_count(), 'affinity': affinity},
        'limits': {
            'nofile': list(resource.getrlimit(resource.RLIMIT_NOFILE)),
            'nproc': list(resource.getrlimit(resource.RLIMIT_NPROC)),
        },
        'tmp': {'path': tmp, 'fstype': mount_fstype(tmp), 'free_gib': round(free / 2**30, 2)},
        'cwd': {'fstype': mount_fstype(os.getcwd())},
    }


def flatten(node, prefix=''):
    if not isinstance(node, dict):
        return {prefix: node}
    out = {}
    for key, value in node.items():
        out.update(flatten(value, f'{prefix}.{key}' if prefix else key))
    return out


def main():
    parser = argparse.ArgumentParser()
    sub = parser.add_subparsers(dest='cmd', required=True)
    cap = sub.add_parser('capture')
    cap.add_argument('--out', required=True)
    dif = sub.add_parser('diff')
    dif.add_argument('left')
    dif.add_argument('right')
    args = parser.parse_args()

    if args.cmd == 'capture':
        with open(args.out, 'w', encoding='utf-8') as fh:
            json.dump(capture(), fh, indent=2, sort_keys=True)
        print(f'wrote {args.out}')
        return 0

    with open(args.left, encoding='utf-8') as fh:
        left = flatten(json.load(fh))
    with open(args.right, encoding='utf-8') as fh:
        right = flatten(json.load(fh))
    for key in VOLATILE:
        left.pop(key, None)
        right.pop(key, None)

    rows = [(k, left.get(k, '--'), right.get(k, '--'))
            for k in sorted(set(left) | set(right)) if left.get(k) != right.get(k)]
    if not rows:
        print('no drift in the recorded facts')
        return 0
    width = max(len(row[0]) for row in rows)
    for key, lval, rval in rows:
        print(f'{key:<{width}}  {lval!r:>26}  {rval!r:>26}')
    print(f'\n{len(rows)} field(s) differ')
    return 0


if __name__ == '__main__':
    sys.exit(main())

Running it on both hosts

Capture on the host you trust first, then on the host you are about to blame, using the exact same command in both places. Copy the two JSON files into one directory so the diff runs on a single machine with nothing else in the picture.

python drift.py capture --out local.json
# on the other host, same command, same interpreter path
echo $VIRTUAL_ENV && which -a python python3
python drift.py capture --out remote.json
python drift.py diff local.json remote.json

The which -a line matters more than it looks, because it catches the case where the two hosts run different interpreters under the same name. That check alone has saved me from arguing about library versions that were never the problem.

Reading the diff

Here is the shape of a diff, and again, these are placeholders rather than measurements of any provider's machine:

time.utc_offset_s                  0                    -25200
encoding.stdout                'utf-8'                  'ascii'
limits.nofile        [1024, 1048576]                 [256, 256]
cpu.affinity                       8                        2
tmp.fstype                     'ext4'                   'tmpfs'

5 field(s) differ

Five rows like that give you five concrete hypotheses in the time it takes to read one stack trace. It could be timezone-bucketed output, a pipe that cannot encode non-ASCII, a connection pool larger than the limit, worker sizing tuned to cores, or a temp file that vanishes on reboot.

Turning the diff into hypotheses with a free model

Small diffs are a good fit for a free model, because the input is tiny and the output is a short list of guesses you can test. The remote side of this comparison is also where a free server option fits: you run the same capture command there without provisioning anything permanent.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I have been using MonkeyCode's free model access as a second reader for the diff output, and the prompt does all the work. Paste the diff and ask for hypotheses with commands attached:

Here is a diff of environment facts between host A (working) and host B (failing). For each differing field, give me a numbered hypothesis in one sentence, the exact shell command that confirms or rejects it, and the output that would count as confirmation. Do not propose a fix before the confirming command has been run. Ignore fields that cannot affect a batch job that reads CSV and writes dated files.

The rule that keeps this honest is simple: a hypothesis is not evidence, and every item must become a command before it becomes a patch. Without that rule you are just trading one plausible story for another plausible story, which is how the first forty-eight hours disappeared.

Decision table: which drift breaks which job

I keep this table next to the diff output, because it turns a wall of differences into a ranked list. Most diffs have one row that matters and four that do not, and the table is how you notice that in seconds.

What I would repeat, and what I would drop

What I would repeat:

• Capture both sides before the failing run, not after, so the facts describe the state that actually failed.
• Store the JSON next to the job output, because the next person to look will be you in three weeks.
• Pin the interpreter with an explicit path in every command you record, including the capture command itself.

What I would drop:

• Adding more print statements to the failing path before the facts are on the table.
• Comparing library versions first, which is almost always the second question rather than the first.
• Trusting a chat answer that arrives without a verifying command attached to it.

Limitations and who should skip this

The snapshot is a point-in-time fact set, so if your remote host is recreated per session you need to capture on every run, not once. resource is POSIX-only, so on Windows you get a smaller fact set and should reach for psutil instead. This harness says nothing about package versions, so pair it with a lockfile hash, and it says nothing about network or DNS behaviour.

If you already run every job inside a pinned container image with a proper CI pipeline, you have solved this problem better than a diff will, so skip it. The same goes for anyone who needs reproducible performance numbers or regulated retention of environment records, because a JSON snapshot is not an audit trail.

The ten-minute cost

Two days of confusion against a harness you write once and run in ten minutes is not a close comparison. Capture both hosts, paste the diff somewhere small, and make the model earn its keep by attaching a command to every guess. If you want to try that second-reader step, the free model access is plenty for a twenty-line diff, and the free server option is enough to get the other side of the snapshot without buying anything.

What I would really like to know is which row of that table has cost you the most hours, because I suspect it is not the one I keep blaming.