BLACKBOX：為事故應變建立記憶

Production incidents rarely arrive as isolated problems. A single alert can be connected to a recent deployment, a downstream dependency, an overloaded database, an unfamiliar service owner, or a similar incident from months ago. Th
e challenge for on-call engineers is not just finding an answer. It is connecting all of that context quickly and explaining why the answer is trustworthy.

That is the problem we built BLACKBOX to solve.

BLACKBOX is an AI-powered incident investigation agent for SRE and on-call teams. It uses FalkorDB as a persistent operational knowledge graph, connecting services, dependencies, alerts, deployments, runbooks, people, and previous postmortems in one place.

From alerts to explanations

When an incident begins, BLACKBOX does more than search through isolated records. It traverses the graph to understand how the alert is connected to the rest of the system.

For example, an alert about high latency in notifications-api may be related to:

• An exhausted connection pool
• A recent pull request changing database pool configuration
• A deployment that introduced the problem
• Services that depend on notifications
• The engineer who owns the relevant code
• A previous incident with a similar failure pattern

The agent follows these relationships to identify a likely root cause, determine the blast radius, find relevant experts, and produce an explanation that an engineer can act on.

This makes the investigation more useful than a simple list of search results. The answer includes the path through the system that led to the conclusion.

FalkorDB as operational memory

The most important part of BLACKBOX is that the investigation does not disappear when the incident ends.

After analyzing an incident, BLACKBOX writes structured findings back into FalkorDB. These include root causes, affected services, resolutions, owners, and postmortem knowledge. Future investigations can then use that information as context.

This creates a feedback loop:

1. An alert triggers an investigation.
2. The agent traverses connected operational data.
3. It identifies findings and recommends a resolution.
4. The investigation is written back to the graph.
5. A future incident can recall that knowledge.

The first investigation may need to explore many relationships. A later investigation with a similar signature can start from the stored context and reach a useful answer more quickly.

Why a graph?

Operational systems are inherently connected.

A service depends on another service. A deployment changes a service. A pull request belongs to an engineer. An incident affects multiple customer-facing systems. A runbook describes how to recover a component. A postmortem explains what happened previously.

These relationships are difficult to preserve in flat documents or isolated tables. FalkorDB gives BLACKBOX a natural representation for them and allows the agent to ask multi-hop questions such as:

• What services depend on the alerting service?
• Which recent deployment is connected to this failure?
• Who owns the code involved in the suspected root cause?
• Which customer-facing systems are in the blast radius?
• Have we seen a similar incident before?
• What resolution was used last time?

Because the results come from connected graph paths, BLACKBOX can explain not only what it believes, but how it reached that conclusion.

Cold and warm investigations

BLACKBOX demonstrates the difference between an investigation with no prior memory and one that can reuse previous knowledge.

A cold investigation starts from the alert and explores the graph to discover relevant context. It may need to inspect dependencies, deployments, code changes, people, and runbooks.

A warm investigation uses the postmortem and findings written during the earlier run. It can recall what was learned, reduce unnecessary tool calls, and focus on confirming the known pattern.

The interface makes this process visible in real time. Users can see the graph traversal, the agent’s tool calls, the findings being written, and the final explanation.

Built for real-time collaboration

BLACKBOX includes a live investigation interface designed for incident response. As the agent works, the frontend streams its progress and updates the graph visualization.

The UI shows:

• The active alert
• The causal path through affected services
• Related deployments and pull requests
• Customer-facing blast radius
• Relevant owners and responders
• Agent reasoning events
• Findings and postmortem write-backs
• Cold-versus-warm investigation results

This visibility matters during an incident. Engineers need to understand what the system is doing, not wait for an opaque answer with no evidence.

The role of the language model

The language model helps select tools, interpret graph results, summarize evidence, and produce a clear incident explanation. However, the graph remains the source of operational truth.

BLACKBOX also includes a deterministic planner fallback. If the model is unavailable or a request exceeds the configured tool budget, the system can still run the core investigation workflow from the graph. This keeps the demo and the incident workflow resilient while preserving explainable findings.

What we learned

Building BLACKBOX reinforced an important principle: agent memory is most useful when it is structured and connected to the world the agent operates in.

A conversation transcript alone does not tell us which service caused an incident, who owns it, or what systems depend on it. A graph can preserve those relationships explicitly and make them available to future investigations.

FalkorDB made it possible to combine persistent memory, multi-hop reasoning, and real-time operational workflows in one system.

Looking ahead

The next steps for BLACKBOX would be to connect it to live observability, ticketing, deployment, and chat systems. That would allow the graph to stay current as services change and incidents evolve.

We would also like to add stronger access controls, tenant isolation, rate limiting, and richer query observability for production use.

The long-term vision is an incident-response system that becomes more useful with every investigation—one that helps engineers move from alert to understanding, and from understanding to action.

BLACKBOX is built around a simple idea:

Every incident should make the next incident easier to solve.