Baize 如何讓長時間執行的智能體任務在重啟後續行並避免重複執行

An agent Run is not a single HTTP request. It can:

• call a dozen tools in a row and run for several minutes;

• hit a write action that needs human approval — and that wait can last hours;

• be triggered by a schedule, e.g. a daily 9 AM summary;

• after grouped subtasks are enabled, spawn a bounded number of group-scoped child runs (one level only), isolating the tool surfaces and approval flows of different business systems;

• after finishing, push the result back to IM or call back a business system.

Along this chain, processes restart, scale out to multiple replicas, downstream APIs time out, and browsers disconnect. If a Run were just a goroutine in memory, a pending channel, then any hiccup would either lose the task or run it twice.

Baize handles this class of runtime reliability in the runtime layer. The core idea is one sentence: make a Run a first-class row in the database, not a goroutine in memory. The queue and workers can restart at any time or scale out freely; "no loss, no duplicate" is guaranteed by database state, leases, and a few background daemon loops.

The numbers here (8 workers, a 60s lease, etc.) are current defaults and are configurable; the mechanisms do not depend on any specific cloud vendor or middleware.

1. Every trigger becomes a Job; the Run row is the source of truth

All trigger sources — chat, webhooks, IM channel messages, schedules, human approvals, child-run receipts — are unified into a single Job enqueued into the queue. Think of it as a work order asking the runtime to do something. There are three job types:

At the same time, every Run has a row in the primary datastore, and the state machine can be read intuitively as:

(occasionally) queued → running → succeeded / failed / cancelled / rejected
running ⇄ waiting for external signal

A newly created Run from chat usually goes straight to running; queued mostly appears in crash recovery and re-enqueue scenarios. The important state is the middle one: waiting for external signal (internal name waiting_human). Whether a write needs approval, a downstream login is required, or a parent is waiting for a child receipt, the Run enters this single suspended state; the reason (approval / login / child receipt) is stored in the suspension payload instead of being split into many incompatible state names.

Workers pull Jobs off the queue with a default concurrency of 8. The queue, event bus, rate limiter, primary datastore, and blob storage all go through swappable drivers: implement the unified interface, register it, and change the driver name in config. The built-in drivers are:

• Primary datastore: SQLite by default, plus Postgres and in-memory;

• Queue / event bus / rate limiter: in-memory by default, Redis for multi-replica setups;

• Blob: local files, S3, in-memory.

The architecture is not tied to any particular database or message middleware. If a team already runs MySQL, Kafka, etc., they can extend the same driver contract without touching the orchestration core — the built-in implementations are "a subset that works today," not an upper bound.

One design tradeoff runs through the whole article: the queue only carries delivery; it is not the source of truth. The Run row in the primary datastore always is. All recovery logic reduces to "query the DB, change state, then re-enqueue."

2. Lease + heartbeat: how one Run executes exactly once across replicas

After scaling out, the same Job may be visible to multiple replicas at once. Baize uses a database lease for atomic claiming, not an in-process lock in the worker. Before actually starting, the executor tries to "lease" the Run, semantically equivalent to:

UPDATE runs SET lease_until = now + ttl
WHERE id = ?
  AND status IN ('queued','running','waiting_human')
  AND (lease_until IS NULL OR lease_until < now);

The update only matches when the lease is empty or already expired, so only one claim succeeds. Default parameters:

• Lease TTL is 60 seconds;

• During execution, a heartbeat renews it roughly every 20 seconds (one third of the TTL), proving "I'm alive and still running";

• The lease is released when execution finishes;

• If claiming fails (another replica holds a valid lease), the worker skips it and does nothing.

Result: even if the same Run is delivered to multiple replicas due to re-enqueue, only one actually executes it.

3. Reconciler: orphan Runs auto-resume after a worker crash

When a worker crashes, the Runs it claimed may still be stuck in running, with the lease still under its name. A reconciler covers this.

On startup it runs one scan immediately, then scans roughly every 15 seconds. Each round:

1. Finds orphan Runs still in queued/running whose lease has expired (or whose creation is more than about 30 seconds old and unclaimed);

2. Re-enqueues a "normal run" job for each orphan;

3. Handles about 100 per batch, leaving the rest to the next round.

The 30-second grace window avoids catching normal Runs that were just created and not yet claimed, reducing false positives.

Note: Runs stuck in "waiting for external signal" are not treated as orphans and forcefully re-run — they are waiting for approval, login, or a child receipt and go through dedicated resume jobs.

Why is there no double execution? Two layers:

• A healthy running Run keeps heartbeating its lease, so the reconciler never picks it up;

• Even if something is re-enqueued in an extreme case, the execution entry has three gates: already in a terminal state → skip; waiting on a human but received a normal-run job → skip; failed to acquire a lease → skip.

Effect: after a process crash, the task usually resumes from its DB breakpoint within one reconcile cycle, without double-starting.

4. Shutdown doesn't kill in-flight work; approvals survive restarts

What happens to in-flight Runs during a deploy restart?

Each Job runs on an independent context, deliberately not tied to the "pull new jobs" lifecycle. On shutdown, the worker stops taking new work first and allows about a 5-second grace window: short jobs finish if possible; Runs long-paused on approval may lose their in-process wait goroutine, but the row in the primary DB still says "waiting for external signal." After restart, a cold-recovery path reconnects, rather than depending on a goroutine that lives forever.

Human approval is the most extreme wait in long tasks — it can span hours or days. Baize covers it with two paths:

• Same-process hot path: while someone is still waiting for this Run on this host, an approval wakes it directly without re-acquiring a lease;

• Cross-restart cold path: the process has changed and the in-process waiter is gone; the approval is re-enqueued as an "approval resume" job, which acquires a lease and continues from the suspended state in the DB.

After grouped subtasks are enabled, the parent also enters "waiting for external signal," with the reason being "waiting for a child receipt"; when the child finishes, a "child receipt" job wakes the parent. Child runs are an optional, one-level-only capability, with a concurrency cap on active children per parent (the current implementation allows at most 2 active child runs) — meant to isolate different business systems' tool surfaces and approvals, not to grow into an infinite fan-out of generic sub-agent trees.

Also, when entering a human wait, the worker releases its slot: it no longer holds a worker slot idle, leaving concurrency for tasks that can still make progress; after the human approves, the run cold-resumes.

Every job is wrapped in panic recovery on the outside: success, failure, and errors all ack the message, so a poison message never jams the queue.

5. Frontend disconnect doesn't lose the stream: numbered, replayable events

A long task is streaming: thoughts, tool calls, and incremental text are continuously emitted. Backgrounding a tab or network jitter can drop the SSE connection.

Baize writes every event to an event table, and each frame pushed to the frontend carries a monotonically increasing sequence number. On reconnect:

• It prefers the standard Last-Event-ID header, and also supports ?after=N;

• It replays the missing events in order from that sequence number;

• When the Run reaches a terminal state, one final end frame is emitted.

Between replicas, a lightweight event bus carries only a wake-up nudge ("this Run has new events") — never the payload. The payload is always re-read from the primary DB by sequence number, so any replica can continue serving the same Run. A lost nudge is harmless: reconnect or periodic fallback fetch still fills in by sequence number.

Refreshing the page, reconnecting, or even landing on a different replica — the process and results are never lost.

6. Scheduled tasks: periodic triggers without duplicates or missed cycles

Chat scheduled tasks (once / daily / weekly) are essentially unattended long-task triggers. Each task is persisted in the DB with state (active / paused / completed), the next fire time, and a placeholder field for "which run is currently in flight."

The schedule scan runs roughly every 15 seconds; on startup it first clears stale "placeholder intermediate states" left by an abnormal shutdown, so a fake placeholder never blocks later cycles. Each round does three things:

1. Release terminal placeholders: if the run pointed to by the placeholder has finished, free the slot, so a long-completed Run never blocks future cycles forever;

2. Advance periodic time: for daily/weekly tasks, even if the previous round is still running, once the next time point arrives, advance "next fire" forward first — avoiding a backlog storm after a long task ends;

3. Atomically claim due tasks: use a conditional update to grab a task that is "due and idle," then start a new Run. Once-only tasks are marked completed; periodic tasks write back the next time.

The supporting fallbacks also live in the DB: failing to start a Run releases the placeholder and records the error; times are computed in IANA time zones; each account has an active-schedule cap (default 20). Atomic placeholders, terminal release, and startup cleanup work together to prevent duplicate triggers at the same moment, and to ensure a crash never permanently skips a later cycle.

7. Result delivery: the transactional outbox and backoff

When a Run finishes, the result must be pushed back to IM or a business webhook. If the process crashes between "generating the reply" and "sending the HTTP request," or the downstream returns a temporary 5xx, the message must not be lost.

Baize uses a transactional outbox for channel outbound: it first writes the message to an outbox table keyed by an idempotent delivery key, then a separate deliverer sends it — generation and delivery are decoupled.

The deliverer pulls a batch of due messages every second (or when woken), about 20 per batch:

• A 2xx response → marked delivered;

• Network errors, 429, 5xx → exponential backoff (1s, 2s, 4s… capped around 60s) then retry;

• Non-retryable errors, or exceeding the max retry count → dead letter, resettable manually in the UI to redeliver.

Large objects like images and attachments are written to Blob first (local files by default, switchable to S3; also a swappable driver), and the outbox stores only the reference — never stuffing binary into the queue table. Business-side webhook notifications have the same reliable delivery and backoff. HTTP plugin callbacks write their events back to the Run's trail through the event-persistence path, which is a separate line from the channel outbox — both emphasize "persist first, then send outward," but they should not be mixed into the same box.

8. The engine layer also holds context for long tasks

So far this is the runtime: don't lose tasks, don't double them, don't block the process while waiting. Long tasks have another side — long conversations, huge tool results, occasional model timeouts — which is handled mainly in the execution engine, orthogonal to the queue/lease and equally transparent to business code:

• When turns pile up, do rolling summaries, and optionally project "key points of the long conversation" (only changing what the model sees as input, not the original messages);

• Especially large tool results or pastes can be offloaded to the session workspace, so the model side only gets a preview and a path, reading back segments on demand;

• Model-call failures retry and fall back; tool timeouts can be split; read-only tools in the same turn run in parallel;

• Human waits release the worker slot (see section 4), so one approval doesn't fill all concurrency.

The two layers fit together as: the runtime guarantees "this task still exists and can resume"; the engine guarantees "it can resume, and the context holds."

9. One table: failure scenarios and their guards

10. Reliability lives in the runtime

These mechanisms are mostly transparent to the people writing agents and configuring skills on top: they face chat, tools, and approvals, without having to write their own retry, recovery, and dedup.

A single machine can use the default SQLite and get "restart-resumable, wait-continuable" without first standing up a full middleware stack. When you need to scale out, switch the queue/event bus to the existing Redis driver, switch the primary DB to Postgres, or extend the driver contract to the team's own databases and messaging systems — the same execution code still holds.

For team-facing agents, the real dividing line is rarely whether a single "think → choose tool → execute" loop works. It's this: when the agent is attached to real business systems, must wait for human approval, must run on a schedule, and must push results back across IM — after a process restart or a network hiccup, are the results still deterministic? Keeping these concerns in the runtime layer is what lets the upper layers focus on the business itself.

Baize is a team-facing, self-hosted AI Agent runtime: a single Go binary, MIT-licensed.

• Source: https://github.com/rebornace/baize

• Docs: https://rebornace.github.io/baize

More runtime internals will follow, including a replayable evaluation corpus.