On-call Brain 以服務依賴圖協助值班工程師排查事故

The newest deploy is almost never the cause. So I gave on-call a graph. (On-call Brain, Graph Hacks by WeMakeDevs x FalkorDB)

It's 3 am. Your phone says checkout p99 latency above 2s. You open the deploy log and look at the newest change: a search deploy from eight minutes ago. You roll it back. Nothing changes.

The real cause was a config change on the payments gateway forty minutes earlier. Checkout calls payments, payments calls the gateway, and someone cut its connection pool from 200 to 20. A list of deploys can't tell you that. A graph of how your services connect can.

That became On-call Brain, my entry to Graph Hacks by WeMakeDevs and FalkorDB.

Click to watch the 4-minute demo: the web app, Slack, Claude Code over MCP, and the graph in FalkorDB Browser.

TL;DR

• Three agents (triage, investigator, comms) work every incident over one FalkorDB graph of the company: services and their dependencies over time, owners, on-call rotations, changes, docs, tickets, chat and past incidents.
• Every answer comes with proof: the evidence path, the suspects ruled out and why, and the Cypher behind each step, with FalkorDB's own timing and query plan.
• When a human confirms a cause, it becomes memory, and the next similar incident is solved in half the steps.
• It's an MCP server first (33 tools), with a Claude Code plugin, Slack and an alert webhook, plus a web UI. Live at https://13-127-186-74.sslip.io/, with every graph on FalkorDB Cloud.
• On 200 generated incidents it finds the planted cause 186 times. A plain change log finds it zero times.

Product thinking: who is this for?

The user is the engineer on call, half awake, with one question: what do I do right now? That shaped every decision:

• Answer first, proof one click away. The screen leads with the cause, how sure it is, who is paged and the fix. "Show me why" holds the graph and the queries for when you want to check.
• Go where they already work. On-call engineers live in their editor, Slack and their alerting tool, so the core product is an MCP server. The web page is one window onto it.
• Humans close the loop. The agents propose; only a person can confirm a cause. That confirmation is what the system learns from.
• Be honest about confidence. "Very likely: 4 of 5 suspects ruled out" says why, and a cause the graph can't support is labelled "needs a human" instead of guessed.

Architecture

Every graph tool is defined once, in one registry, and served three ways: to the agents, over MCP and over HTTP. The agents never call each other. They read and write one shared incident board in the graph, so any agent can be restarted mid-incident and pick up where it left off. Claude Opus 5.5 chooses each step and writes a one-line reason; without an API key, a deterministic planner does the same job, which is also what the tests run against.

FalkorDB is the only database. Besides one graph per company, it holds a coordination graph (locks, rate limits and the LLM budget) and two libraries of functions that run inside the database. More on why below.

The graph model

The most important design choice is that edges carry time. DEPENDS_ON, OWNS and ON_CALL all have validity windows, and nothing is ever deleted. When checkout stopped calling legacy-billing three days ago, that edge got a valid_to. Every query asks what was true at the moment of the alert. Each company gets its own graph in FalkorDB, so memory and knowledge never mix between tenants.

Facts carry a second time too: recorded_at, when the graph learned them. If CI reports a change after the incident opened, the agents still use it, and the answer lists it as "learned late" instead of pretending it was known all along.

The query that does most of the work

Finding suspects is a temporal multi-hop walk: start at the alerting service, follow up to six dependency hops, keep a path only if every edge on it was live at alert time, then take the recent changes on the services reached, nearest first. In plain Cypher:

MATCH (i:Incident {id: $incident_id})-[:TRIGGERED_BY]->(a:Alert)-[:FIRED_ON]->(o:Service)
MATCH p = (o)-[:DEPENDS_ON*0..6]->(s:Service)
WHERE all(r IN relationships(p)
          WHERE r.valid_from <= a.at AND (r.valid_to IS NULL OR r.valid_to > a.at))
WITH a, s, min(length(p)) AS hops
MATCH (c:Change)-[:CHANGED]->(s)
WHERE c.at >= a.at - $window AND c.at <= a.at
RETURN c.id AS change, s.id AS service, hops
ORDER BY hops, c.at DESC

That version has a cost: the WHERE filters paths after FalkorDB has expanded them, so the walk still wanders into every retired dependency's subtree. The product version flips it around. It starts from the few changes in the window (a range index on Change.at), then calls a JavaScript function that runs inside FalkorDB and walks only live edges, stopping as soon as it has reached every changed service:

WITH o0, hits, [o IN [o0] + folded |
     {o: o, reach: oncall.live_reach(o, alert_at, 6, [h IN hits | h.s.id])}] AS walks

A test checks that it returns exactly the same rows as the plain form. It's as fast at 1,000 services and 2 to 4 times faster at 2,500.

Each suspect is then checked: is there a live path from the alert to its service, and did errors appear along it, in order, after the change? For the first incident:

Legacy-billing is the interesting one: it deployed and threw errors right before the alert, so any rule that watches errors blames it. Only the retired edge rules it out.

Memory you can measure

Confirming a cause writes CAUSED_BY and a runbook back into the graph. Three days later, refunds slows down. Recall is structural, not textual: it finds resolved incidents whose cause sat on the new alert's live dependency path. The checkout incident matches, because the same gateway sits upstream of refunds, and its fix is tested first. For a fair comparison the same alert also runs on a copy of the graph that never saw the first incident:

Three alerts, one incident

Checkout, refunds and subscriptions alert within four minutes. Triage projects the alerting services and the upstream services that went red into a scratch graph and runs WCC (weakly connected components) to decide which alerts are one incident. A postgres-orders change explains all three and leads; a subscriptions deploy explains only one, so it's kept as "also possible" rather than thrown away. Criticality for paging comes from PageRank on a point-in-time projection of the live dependencies.

The company brain

Docs, decision records, postmortems, tickets and chat sit in the same graph, linked to the services, changes and people they mention. "Who owns kafka?" returns the team, who is on call now and the sources, with an older decision record flagged as superseded. A follow-up like "and who is on call for it?" works because each conversation is stored in the graph too. No model runs here: full-text search finds the documents, and the graph decides what's current and what to cite.

Beyond the web page

• Claude Code plugin: ask "what breaks if postgres-orders goes down?" and Claude picks the blast-radius tool by itself.
• Slack: /oncall who owns payments-gw? answers inline, and /oncall incident subscriptions latency runs the agents and posts the result with Confirm and Wrong cause buttons.
• Webhook: an alerting tool posts the alert, the agents run headless, and the status update comes back.

Going all in on FalkorDB

Late in the hackathon I moved the live site's graphs to FalkorDB Cloud and went through the docs looking for every place the database could do the work instead of my code. What changed:

The Free tier keeps nothing across a restart, so a watchdog checks once a minute and puts back the constraints, the functions and the demo company.

Running it in public

The app runs on one ARM instance on AWS, in Docker behind Caddy for HTTPS, with no SSH and secrets in SSM. The graphs live on FalkorDB Cloud in the same region. A public demo with a real model behind it needs guardrails: per-client rate limits, a cap on concurrent runs, daily and hourly token budgets (past them the rules planner takes over and says so), request size limits, security headers, and a token on every outside write. Every issue the security scans found is fixed.

Does the graph actually matter?

An eval of 200 generated incidents, each with a planted cause zero to three hops upstream, decoys of every kind, unrelated errors, silent hops and cascade victims:

Design decisions and trade-offs

What I learned

• Model time explicitly. Retiring an edge instead of deleting it was the single most useful decision.
• Let the database enforce safety. Every read runs as GRAPH.RO_QUERY, so a read tool can't write even if its query is wrong, and constraints refuse bad data whoever writes it.
• Independent review catches what the builder can't. Every round of separate code, security and design review found real issues, including two-minute hangs under load.
• FalkorDB details worth knowing:
  • shortestPath works in WITH and RETURN, not in MATCH.
  • A variable-length bound can't be a parameter (*1..$depth fails), so the hop limit is a function argument.
  • A MANDATORY constraint is checked the moment the edge is created, before any SET in the same query, so the required property goes inside the MERGE pattern.
  • Create constraints only when they're missing, never while writes are running.
  • On 4.22, two GRAPH.UDF LOAD REPLACE calls close together under traffic can crash the server. Load a library once, and replace it only when its code changed.
  • FalkorDB Cloud allows a fixed list of commands. Design for graph commands only and it just works.

What I'd improve next

• Tell "red because of this change" apart from "red at the same moment for another reason": most of the eval's misses are that case.
• A path for incidents with no change behind them, such as pure capacity problems.
• Real ingest from CI/CD, monitoring and chat into the write tools that already exist (record_change, record_doc, record_message, record_ticket).
• Per-user auth on the public API, and runbook confidence that decays with age.

Try it

• Demo video: https://youtu.be/eriObL8LlsQ
• Live: https://13-127-186-74.sslip.io/ (press "Simulate an incident")
• Recorded runs, no server: https://harz05.github.io/oncall-brain/
• Code, graph model and every Cypher query: https://github.com/harz05/oncall-brain

Built with Python, FastAPI and FalkorDB, with Claude Opus 5.5 planning at runtime and Claude Code as my coding assistant. Thanks to WeMakeDevs and FalkorDB for Graph Hacks.