智能體寫入後崩潰，應否重試？本地恢復實驗拆解回執判斷

The hardest moment in an automation often sits between two acknowledgements: the target accepted the change, but the executor has not recorded completion. Restarting the agent restores process availability; it does not resolve uncertainty about the previous effect. In this lab, I terminate the process in that exact interval and inspect what survives. The next decision depends on persistent evidence, not the model's confidence.

What was verified

• Implementation: lab-security-remediation, tag v0.4.0; local tests executed on October 27, 2026.
• Observed result: 22 tests passed. In the receipt exercise: version 2, one receipt, and no second change after recovery and resubmission.
• Scope: Local SQLite, synthetic target, simulated review. No credentials or AWS calls required.
• Not demonstrated: Does not establish exactly-once execution across remote APIs, power-loss resilience, or production-ready authorization.

Start with the question the log cannot answer

Imagine a tool that fixes a configuration. It receives a reviewed proposal, checks the target version, and applies the change. Before returning success, the process terminates. The client sees an interruption. The target may already have changed. A log line saying the call started cannot distinguish those possibilities.

In this exercise, the target is only a SQLite row representing a synthetic bucket. The blockPublicAccess property is a teaching device; this is not an S3 call. The proposal identifies the operation and the expected version. An intent record, called the ledger, stores the reviewed request. A receipt records that this request produced this target version.

That separation creates a testable question: what evidence connects this operation to this change? Seeing the desired configuration does not tell us who changed it. Finding a restarted process does not tell us whether its predecessor finished. A client error describes the client's observation, not necessarily the target's reality.

Where evidence must survive

Local lab architecture. The tables share one SQLite file, but intent, effect, and completion use separate transactions. The target update and its receipt are atomic with each other.

Local CLI

• Reviewed executor (compute)
• Recovery reader (compute)

SQLite · separate commits

• Operation ledger (storage)
• Target + receipt (data)
• Audit events (storage)

Flows

• executor -> ledger: INTENT
• executor -> target: EFFECT
• target -> recovery: EVIDENCE
• ledger -> recovery: REQUEST
• recovery -> ledger: CLASSIFIES
• recovery -> events: DECISION

Prepare a run another person can reproduce

Use Node.js 22.13 or later with node:sqlite available. The tagged source is the reference for this exercise; do not assume the published npm version is identical. Installing dependencies is not required to run this lab. Clone into a new directory, check the tag, and run the tests before changing a scenario.

Choose fresh database names for each complete repetition. A reused database retains history, and that history changes the expected result. Preserve the original proposal too: generating another after the change can produce a different digest for the same identifier and should cause a conflict, not implicit permission to execute again.

The intentional failures are separate from inspection commands. In a script using set -e, explicitly capture exit code 86 so the expected termination does not end the investigation. The uncertainty exercise returns 2, another expected outcome. These codes are evidence; do not erase them with an indiscriminate || true.

Setup: pinned source and local tests

git clone https://github.com/fernando-moretes/lab-security-remediation.git
cd lab-security-remediation
git checkout v0.4.0
node --version
npm test

Experiment 1: the effect exists; the acknowledgement was lost

Create the database and export the proposal. Running with --approve simulates explicit review and records intent. The after-target crash point terminates the process after the commit that updates the target and writes its receipt, before marking the operation complete. This is not an exception caught inside one function: the CLI exits, and subsequent commands open the database in other processes.

After the crash, inspect three things together: operation status, target version, and receipt existence. In the October 27 run, the operation was pending, the target was at version 2 with blockPublicAccess: true, and one receipt matched the proposal digest. That combination demonstrates why retrying solely because the ledger is pending would be an uninformed decision.

Intentional crash: run exits 86; continue with inspect

node bin/cli.mjs durable init ./receipt.sqlite
node bin/cli.mjs durable plan ./receipt.sqlite > receipt-proposal.json
node bin/cli.mjs durable run ./receipt.sqlite receipt-proposal.json --approve --crash after-target
# Expected exit: 86
node bin/cli.mjs durable inspect ./receipt.sqlite
node bin/cli.mjs durable recover ./receipt.sqlite synthetic-finding-001
node bin/cli.mjs durable run ./receipt.sqlite receipt-proposal.json --approve
node bin/cli.mjs durable inspect ./receipt.sqlite

Recovering the record does not require repeating the effect

The recover command reads the operation, receipt, and current state. Here it changes the classification to reconciled and records the decision in the event history. It does not mutate the target. Resubmitting the original proposal also returns reconciled, with retryable: false. In the final inspection, the version remains 2 and the receipt count remains 1.

Those are the invariants worth checking. The test is not simply searching for the word success in the output. Compare the observable effect before and after recovery. Compare it again before and after resubmission. If the version increased again, an apparently correct status would hide a second execution.

A receipt is historical evidence, not a permanent health certificate. If the target changes later, the earlier operation's history can remain valid while current state diverges. The implementation exposes driftAfterReceipt to distinguish those questions. Restoring a desired configuration, when necessary, requires a new reviewed decision about current state.

Observed results from both experiments

Experiment 2: the state looks right; ownership is unknown

Now use another database. Interrupt execution at after-intent, before applying the effect. Then use the teaching command drift to change the target through another path. During recovery, the desired property is true and the version is 2, just as in the first experiment. The decisive difference is the missing receipt.

The observed result is uncertain, retryable: false, with exit code 2. Resubmitting the original proposal preserves that classification and does not mutate the target again. The system does not infer ownership from matching values. This behavior trades automatic progress for a decision that can be investigated.

Do not delete the operation to unblock a queue. That removes the memory used to recognize duplicates. Preserve the database, proposal, and results; investigate the change and define the next step as a new decision. The lab offers no magic administrative resolution button and should not suggest that manually editing the ledger is a safe operating procedure.

The final configuration matches, but the receipt is missing

node bin/cli.mjs durable init ./uncertain.sqlite
node bin/cli.mjs durable plan ./uncertain.sqlite > uncertain-proposal.json
node bin/cli.mjs durable run ./uncertain.sqlite uncertain-proposal.json --approve --crash after-intent
# Expected exit: 86
node bin/cli.mjs durable drift ./uncertain.sqlite true
node bin/cli.mjs durable inspect ./uncertain.sqlite
node bin/cli.mjs durable recover ./uncertain.sqlite synthetic-finding-001
# Expected exit: 2, status: uncertain
node bin/cli.mjs durable run ./uncertain.sqlite uncertain-proposal.json --approve
# Expected exit: 2; no additional target change

Three commits explain the failure windows

The implementation persists reviewed intent in one transaction. In another, it compares the expected version and writes the target together with its receipt. In a third, it marks completion in the ledger. The receipt and change are atomic with each other; the whole execution is not one transaction. That choice deliberately creates the window we just tested.

BEGIN IMMEDIATE serializes writers to the local file with a bounded contention wait. Storage uses WAL and synchronous=FULL. These choices give the exercise transactional semantics, but process-termination tests are not power-loss, disk, corruption, or network-filesystem tests. The cost of serialization matters too: one local file does not represent distributed scalability.

The suite covers other boundaries: crashes before the target commit, before the completion commit, and after completion; concurrent processes; repeated identifiers with different payloads; unreviewed proposals and state changes. The 22 passing tests check these properties in the lab. They are not an availability percentage or a production agent success rate.

C4

A person runs a recovery lab and inspects synthetic evidence. The model calls no AWS services and requires no LLM.

Containers

• CLI (Node.js): Independent processes for plan, run, inspect, and recover.
• Persistent store (SQLite local file): Operations, target, receipts, and history persisted in one file.

Components

• Proposal validation: Validates the request, simulated review, and expected version.
• Conditional mutation: Updates target and receipt in the same commit.
• Recovery classifier: Reads evidence and records the classification without executing the mutation.

What changes when the tool calls AWS

In a real integration, recording intent in a database and calling a remote API are operations in different systems. The local commit does not encompass the API. The lab deliberately keeps target and receipt together so evidence is inspectable; moving the design to the cloud requires reassessing that boundary.

The AWS Builders' Library explains the value of an explicit client-provided identifier for recognizing retry intent. Check the concrete operation's idempotency contract: scope, token lifetime, behavior with changed parameters, and repeated-call responses. Do not infer a universal guarantee because another service accepts a token.

For each mutable tool, document which remote evidence is available, how long it remains queryable, and who resolves inconclusive outcomes. When the target cannot provide sufficient proof, retain a visible uncertainty state. Adding a queue, an agent framework, or an MCP server does not independently close the gap between effect and acknowledgement.

Separate current authorization from historical evidence too. In this lab, --approve is only a local review simulation. A real service must verify identity, scope, and validity before mutation; revoked approval must not become permission for a new execution just because an old record exists. A recovery read and a mutation attempt require different decisions.

Decision: uncertainty is an operational outcome

Status: accepted

Context

The client may lose the response after a committed effect at the target.

Decision

Use persistent evidence to reconcile; block resubmission when the effect cannot be attributed to the operation. In this lab, recover never mutates the target.

Consequences

• Requires an owner for investigating uncertain states.
• Receipt and proposal retention is part of the recovery contract.
• May require intervention even when current state looks desirable.

Checklist for reviewing your own tool

1. Define intent: Choose a stable identifier for the same operation and an explicit rule for conflicting payloads. Save the original proposal and contract version.

2. Locate the real commit: Mark where the target changes and where acknowledgement is persisted. Inject an interruption between them, away from production resources.

3. Compare effects: Observe version, receipts, and history before and after replay. A success status alone does not demonstrate the absence of duplicates.

4. Exercise ambiguity: Have another actor change the target without the operation's receipt. Verify that matching values do not become assumed ownership.

5. Define retention and operations: Document the evidence retention window, investigation owner, and consistent database-copy procedure. Close processes before copying the lab file.

A Well-Architected reading

• operational-excellence: Turn uncertain into an investigation queue with an owner and minimum evidence. Recovery needs a runbook and an exercise as well as observability.
• security: Local review authenticates nobody. In production, protect the ledger, minimize sensitive data, and revalidate authorization for the next mutation.
• reliability: Test lost acknowledgement, not only unavailability before a call. Separate duplicate recognition, reconciliation, and a newly authorized attempt.
• performance: Measure contention and reconciliation latency in the real architecture. SQLite writer serialization is not evidence of distributed throughput.
• cost: The exercises make no AWS calls. For a remote implementation, estimate evidence storage and recovery queries; do not invent savings from this local test.
• sustainability: Avoiding redundant effects and attempts reduces unnecessary work, but the lab measures neither energy nor emissions. Do not turn that hypothesis into a number.

Traps the exercise makes visible

• Retrying every interrupted call without consulting effect evidence.
• Treating the desired final value as proof that this operation produced it.
• Generating a new proposal after the change and calling it a retry of the original.
• Deleting receipts or ledger records without a policy that preserves the duplicate-recognition window.
• Presenting a local SQLite transaction as an exactly-once guarantee across remote services.

Diagnostic replay and mutation retry are different operations

To share a reproducible case, include the code version, synthetic initial state, proposal, interruption point, exit codes, and before-and-after inspections. A terminal screenshot helps communicate; the files and commands make verification possible. Remove credentials and real data before opening an issue.

The public evidence associated with this article contains results from both scenarios. The invitation is to discuss a concrete contract: which observation permits reconciliation, which permits another attempt, and which requires investigation? A small contribution, such as adding a failure case with an expected outcome, can be more useful than adding another tool to a catalog.

If you want to experiment, run both scenarios in the repository and describe which failure boundary you would test next. To bring this into your AWS architecture, start with one mutable operation and document its idempotency and recovery contract. The test then becomes a property the team can review, beyond a generic agent demonstration.

References

• Observed local results: October 27, 2026 (JSON)
• Source and exercise, lab v0.4.0
• Executable recovery tests
• AWS Builders' Library: Making retries safe with idempotent APIs
• SQLite transactions
• Node.js SQLite API

The question to bring to your review

Before authorizing another attempt, ask for evidence that distinguishes a confirmed effect, an absent effect, and an unknown effect. In this lab, a receipt allows reconciliation without another execution; its absence in the presence of drift requires stopping. The value of the exercise is making that decision reproducible and reviewable by someone else.

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.