Brooks Forsyth 50 次中途終止 LLM Agent，整理 Temporal 恢復機制的實作與限制

An LLM agent is a loop: ask the model, run the tools it asks for, feed the results back, repeat. Every framework ships one, and almost none survive their own process dying. For a chatbot that's fine. For a forty-minute research task or a triage flow that waits on a human, it means a redeploy or an evicted pod throws away the conversation and re-runs every tool, including the ones that sent an email. I wanted an answer I could defend in a security review. Here's what I built, what I tested, and two things that surprised me.

What Temporal gives you, and what it doesn't

Temporal is durable execution: you write a function, the server records every step as event history, and if the process dies another one replays the history to rebuild state and continue. The function must be deterministic, so side effects live in activities that are recorded by result. An agent loop maps onto this almost too neatly: the loop is deterministic, and the two side effects are the model call and the tool call. The whole workflow is about forty lines; messages is workflow state, and a new worker rebuilds it from history.

What Temporal does not give you: it doesn't know your tool already sent the email, it doesn't know a human needs to approve something, and it won't let you change the loop's code while runs are in flight without a non-determinism error. Those three gaps are the rest of this post.

The four things I had to build

1. Crash-resume, with proof. Heartbeats on every activity so a dead worker is noticed in seconds. The test SIGKILLs a worker subprocess mid-tool, starts a fresh one, and asserts the per-activity attempt counts from event history:

 #   activity       attempts   completed
 1   llm_call              1   yes        <- ran once, before the kill; never re-run
 2   tool_execute          2   yes        <- was executing when the worker died; retried
 3   llm_call              1   yes
 4   tool_execute          1   yes
 5   llm_call              1   yes

The assertion isn't "the run finished." It's "activity 1 has exactly one attempt."

2. Approval gates. Mark a tool requires_approval=True and the workflow parks on a server-side timer waiting for an approve or reject signal. The test parks a run, shuts down the only worker, approves from a separate process with nothing serving the queue, starts a new worker, and checks the run finishes with the reviewer's name recorded. Rejections and timeouts go back to the model as an error result with the reason; the tool never executes.

3. Safe upgrades. The one I got wrong. See below.

4. A shared idempotency journal. Every tool call gets a key (workflow_id, turn, tool_use_id) that's stable across retries; the result is written under it before the activity returns, and a retry that hits the journal returns the record without running the tool. Version one was JSON files on local disk, which works for one worker on one machine, which is to say it doesn't. The real one is a Postgres table with INSERT ... ON CONFLICT DO NOTHING.

The failure matrix

The question I'd ask anyone who calls their agent "durable": where exactly can the worker die, and what happens? Here's mine.

Worker dies...	What Temporal sees	What happens on the next worker
during the LLM call	heartbeat stops; attempt times out in ~20 s	LLM call retried. Nothing was recorded, nothing is lost.
after the LLM returned, before completion was recorded	activity never completed	Same. Temporal only trusts recorded completions.
during a tool execution	heartbeat stops; attempt times out in ~10 s	Tool retried. Journal miss (it never finished), so the tool runs again. For a non-idempotent tool this is the one open case.
after the tool finished and was journaled, before completion was recorded	activity never completed	Tool retried, journal hit, result returned without re-execution. This is the case the journal exists for.
between steps (workflow task)	workflow task times out	Replay. No activity re-runs.

Four of the five rows have tests. Row three is the honest gap: a crash during a non-idempotent tool, before the journal write, still re-runs it. Closing that means recording intent before acting and reconciling on retry, which is a different project. I'd rather name the gap than pretend the journal covers it.

Two things that surprised me

patched() is sticky, and that's the right call. Temporal's mechanism for changing workflow code is workflow.patched("id"): new code takes one branch, old histories replay through the other. I added a v2 that summarizes long tool results, guarded it, and wrote a test that starts a run on v1, swaps in a v2 worker mid-activity, and asserts the run picks up the new behaviour from there.

The test failed. The run finished on v2 with zero failed tasks, but every step stayed on the v1 path, including the ones after the swap. The SDK source says why: patched() returns true "if not replaying, or if replaying and has seen this patch before." A run that first meets the patch while replaying an unmarked history is pinned to the old branch for life.

For an agent that's the property you want, and I'd been about to argue against it. A deploy never changes the shape of an in-flight conversation. Run A's context is byte-for-byte what it would have been if nobody had deployed; run B, started seconds later on the same worker, gets v2 from step one. The rewritten test asserts exactly that. Someone at Temporal had thought about this longer than me.

The card gets charged twice, reproducibly. I'd written "the journal prevents double execution" before I'd proven it. So: a tool that appends a line to a file each time it really runs, a worker killed in the window after the journal write but before Temporal recorded completion, a second worker retrying. Per-worker file journal: two lines. Shared Postgres journal: one line, result marked from_journal. Both are tests, and the failing one is kept on purpose. The window is microseconds wide in real life, so a test-only hook holds the activity open after journaling so the crash lands in the right place every run.

What's still open

Row three of the matrix: a crash during a non-idempotent tool, before the journal write. Needs intent-before-action and reconciliation on retry.
The three-deploy rollout for patches (patched → deprecate_patch → remove) is documented but I've only shipped the first step. The replay check (Replayer(workflows=[AgentWorkflow]).replay_workflow(history)) is what gates the second.
Tool schemas are re-sent to the model on every turn. That's a token cost, not a correctness problem, but it's the next thing I'd measure.

All of it, with a test next to each claim: https://github.com/Bforsyth1234/fde-kit. make up starts Temporal, Postgres and a trace viewer; make crash-demo and make upgrade-demo are the two demos; pytest runs 34 tests against a real local Temporal server.

If you have an answer for row three that isn't "make every tool idempotent," I'd like to hear it. If every tool were idempotent that would be great, but we don't live in that mythical world.