跳到正文
DEV Community· Evgenii Iakovenko·· 3 小時前AI 評分55

作者團隊的 API 停擺 56 分鐘,卻沒有觸發錯誤或告警

Our API was down for 56 minutes and nothing threw an error

AI 導讀

作者團隊的平台首頁及登入頁停擺 56 分鐘,連線池上限為 10,並行購買交易互相等待第二條連線,令請求掛起且沒有告警。原因是交易內透過注入的 Repository 查詢設定,另行取得連線;團隊把讀取移至交易前,並將 DB_POOL_MAX 設為 20、連線取得逾時設為 10 s。團隊亦把交易內不得另取連線寫入審查規則;13 天後,另一條資金流程的同類問題在程式碼審查中被發現。

正文

The night nothing threw

In August the homepage and the login page of our platform stopped answering. Requests just sat there: no error page, no timeout, no 5xx, no exception in the API, no alert. The outage lasted 56 minutes, until we noticed it ourselves and fixed it.

The database was up the whole time. Nothing reported a problem while the platform served nothing. That is the part I keep coming back to.

What the platform looks like

It is a consumer platform with real-money wallets. The stack is NestJS and TypeORM on node-postgres (pg-pool), Postgres, Docker Compose, one API process plus a background worker. We were a small team, and new features were prioritized over tech debt, so observability work kept losing to the next release. That is not an excuse. It is the context in which this bug lived without anyone looking at it.

How a purchase took the pool hostage

A purchase runs inside dataSource.transaction(manager => …). Inside that transaction, the step that credits rewards for the purchase read the rewards settings through a service. That service queried through an injected repository, not through the transaction's manager. So every one of those calls acquired a second pool connection while the transaction still held its first. Settings cache misses did the same thing.

The code that introduced the second acquire merged about ten days before the outage.

With 10 or more concurrent purchases, each transaction held one connection and queued for a second one, which only another of those transactions could release. Pool max was 10. Every connection was taken, every acquire waited, and nothing else could get in.

Postgres's deadlock detector never fired. The wait was in the application, not on a lock. From the server's side the sessions were just idle in transaction, holding their row locks, and Postgres has no way to know they were waiting on each other through the pool. Every new request queued behind them.

And it was silent because connectionTimeoutMillis was unset. A pg-pool acquire with no timeout queues forever. No error, no log line, no 5xx. A hanging request is not a failing request, so nothing in our monitoring had a reason to complain.

What I assumed and what was true

Three things I believed that evening, and what was actually going on.

  1. I assumed the pool was 20. It ran at the pg default of 10. The pool settings sat in an extra block under a config namespace (registerAs) that TypeORM never reads. The block looked like config, so that dead block had been hiding a max-10 pool for months.
  2. I assumed the rewards feature flag was off in production, because the environment doc said so. It was on. I checked through the API on the day of the incident. During triage that stale line hid the flag's role: the code path was armed on production while every doc said it wasn't.
  3. I assumed the health endpoint would tell us something. It never touched the database. The worker's healthcheck was an HTTP probe that could never pass.

The fix that night and the trade-off

We shipped the fix about an hour later, the same evening. The tempting move was to bump the pool and go to bed, and I did weigh it that night. I didn't want that as the whole fix, because the pool size was not the bug. A bigger pool only raises the number of concurrent purchases you need before the same thing happens. The main concern was that the shape would come back at a load level we could not predict.

So the structural fix came first: the settings read is hoisted out, prefetched before the transaction opens, and passed in as a required field on the input type. Forgetting the prefetch is now a type error, not a latent deadlock. A second code path with the same shape got the same hoist. The pool also got an explicit max and an acquire timeout, so a starved acquire fails loudly instead of hanging.

class OrderService {
  constructor(private readonly dataSource: DataSource, private readonly settingsRepo: Repository<Settings>) {}

  async place(orderId: string) {
    return this.dataSource.transaction(async (manager) => {
      const order = await manager.findOneOrFail(Order, { where: { id: orderId } });
      const settings = await this.settingsRepo.findOneByOrFail({ key: "wallet" }); // second connection
      return settle(manager, order, settings);
    });
  }
}
type PlaceOrderInput = {
  orderId: string;
  settings: Settings; // required: forgetting the prefetch is a type error
};

class OrderService {
  constructor(private readonly dataSource: DataSource) {}

  async place({ orderId, settings }: PlaceOrderInput) {
    return this.dataSource.transaction(async (manager) => {
      const order = await manager.findOneOrFail(Order, { where: { id: orderId } });
      return settle(manager, order, settings);
    });
  }
}

// caller
const settings = await settingsRepo.findOneByOrFail({ key: "wallet" });
await orderService.place({ orderId, settings });

The fix was +162/−69 in 9 files. The hardening merged about 30 minutes later, +367/−46 in 15 files.

Setting Default What it buys
DB_POOL_MAX 20 the size the pool was supposed to have
DB_CONNECT_TIMEOUT_MS 10 s acquire timeout: fail loudly instead of hanging
DB_STATEMENT_TIMEOUT_MS 30 s API / 120 s worker not applied to the migrations CLI: a long CREATE INDEX must stall, not die
DB_IDLE_IN_TXN_TIMEOUT_MS 60 s API / 300 s worker plus a 60 s server-side backstop on Postgres, for clients that set nothing

All four live in one function, which is now the only place pool config is real. The dead config block was deleted. The health endpoint probes the pool with SELECT 1 and a 2 s budget, returning 503 on failure. The worker healthcheck checks a heartbeat file instead of an HTTP endpoint it could never satisfy. An autoheal container restarts anything marked unhealthy. New gauges expose db_pool_total, db_pool_idle and db_pool_waiting.

And the rule got written down for humans and for AI agents, in the backend rules file and AGENTS.md: never acquire a second connection inside a transaction, with four sub-rules: thread the manager, hoist the read, distrust "it's cached", and no external awaits inside a transaction.

The same class of bug came back 13 days later

Thirteen days later a teammate fixed the same class of bug in another money flow. A settings read inside a balance-reservation transaction was moved onto the caller's manager, so it now runs on the connection already holding the per-user advisory locks. Same shape, different feature, same fix.

This time it was found in code review, not by an outage. For me that is the real result of that night: the pattern now has a name and a written rule, so reviewers look for it.

To summarize

  1. Grep your transactional code paths for injected repositories and the bare DataSource. Anything that isn't manager is a second acquire.
  2. Set an acquire timeout (connectionTimeoutMillis in node-postgres). The default is to wait forever.
  3. Check that the pool size you set is the pool size in use: log pool.totalCount and options.max at boot, or export a gauge. Config can sit somewhere the ORM never reads.
  4. Set idle_in_transaction_session_timeout on the server, not only per client.
  5. A healthcheck that doesn't touch the database cannot detect a database-side outage.
  6. "It's cached" can still be a conditional database read, on a miss or on keys you never cache.
  7. Encode the rule in types where you can. A required prefetched input beats a code comment.

What I'd do differently

New features always won the prioritization, and I understand why. Deadlines were real. But tech debt that never wins a prioritization gets paid as downtime, and I would rather price that in minutes before it happens than explain it afterwards. That is the conversation I would force earlier, in plain numbers: this gap costs us an outage of unknown length, and here is what the hardening costs to prevent it. Getting that agreed with the business up front is the part that was missing, and it is my job to make that ask concrete instead of hoping someone finds the time.

來源:DEV Community · dev.to