構建 TraceOps：教 AI 以圖譜而非猜測調查生產環境事故

How I built an evidence-first AI incident investigator using FalkorDB, Spring Boot, and JevModel for FalkorDB Graph Hacks 2026.

The moment that inspired TraceOps

Imagine this: checkout requests are failing, customers cannot complete payments, and alerts are firing across multiple services.

The monitoring system has detected the problem. But one question remains: What actually caused it?

Was there a recent deployment? Is a downstream service unhealthy? Is the database struggling? Or are several symptoms pointing toward a deeper dependency failure?

In a microservices architecture, answering these questions can require an engineer to investigate alerts, service dependencies, deployment history, logs, and previous incidents across multiple tools. Each source offers a piece of the puzzle, but the engineer must connect those pieces before making a decision.

AI can summarize an alert or suggest a likely cause. However, a convincing explanation is not necessarily a correct one. A deployment that happened two minutes before an incident may be relevant, but timing alone does not prove causation.

I wanted to explore a different approach: What if an AI incident investigator could reason over connected operational evidence, explain why a hypothesis is plausible, and acknowledge when it does not have enough information?

That question led me to build TraceOps.

Introducing TraceOps

TraceOps is an evidence-first AI incident investigator for microservices. It connects incidents, services, dependencies, deployments, and evidence in a graph, then uses that context to help engineers prioritize their investigation.

The goal is not to replace the engineer or automatically declare a root cause. It is to reduce the effort required to move from a noisy incident to a focused, evidence-backed investigation.

The prototype brings together three key ideas:

Graph-native context: Understand how services and operational events relate to one another.
Explainable investigation: Rank plausible causes using available evidence and show the context behind those hypotheses.
AI-assisted reasoning: Use a structured model assessment to help identify likely causes, missing evidence, and useful next checks.

TraceOps is built around a simple principle: an AI system should help engineers investigate with evidence, not replace evidence with confident guesses.

Why I chose a graph-first approach

Traditional incident data is often distributed across different systems. A service alert, a deployment record, and a database health signal might each look reasonable in isolation. The challenge is understanding how they connect.

A graph is a natural way to represent these relationships.

For TraceOps, I used FalkorDB to represent operational entities and their connections. The graph model includes services, incidents, evidence, and deployments, with relationships that describe affected services, dependencies, and where evidence was observed.

Consider a checkout service that depends on a payment service, which in turn depends on a payment database.

If checkout begins failing, investigating checkout alone may not be enough. A signal from the payment database could provide an important lead, even if the initial alert appears at the checkout layer.

Graph traversal allows TraceOps to explore connected dependencies instead of treating the affected service as an isolated component.

I also included deployment context because recent changes can be useful investigation signals. However, TraceOps must distinguish between a relationship worth investigating and a cause that has actually been established.

That distinction shaped the design: the graph supplies connected context, while the investigation logic interprets that context to prioritize hypotheses.

Under the hood: how TraceOps works

I chose Spring Boot for the backend because I wanted an explicit, structured investigation pipeline rather than a collection of loosely connected AI calls.

The architecture has three main parts.

1. FalkorDB: connecting the evidence

FalkorDB stores the graph used by the investigation workflow. Cypher queries retrieve incident context, related evidence, deployment information, and service dependencies.

This lets the application examine the incident in relation to the wider system. Instead of passing an isolated error message to an AI model, TraceOps can first gather relevant context from the graph.

2. Spring Boot: orchestrating the investigation

The Java backend exposes REST endpoints for incident scenarios, investigation results, and AI assessments.

The investigation service retrieves the incident context, explores service dependencies, and ranks hypotheses using explicit rules. These include possible dependency degradation, deployment regression, and insufficient evidence.

The rules are deliberately understandable. When a hypothesis ranks first, its score represents a heuristic ranking signal, not a statistically calibrated probability.

I wanted the first version to be testable and predictable before making the workflow more sophisticated.

3. JevModel: adding structured AI assessment

TraceOps also integrates JevModel to assess the investigation context.

The backend sends structured information about the incident, available evidence, deployments, dependencies, and heuristic hypotheses. The model returns a structured assessment of a potential cause and associated confidence information.

The AI assessment complements the graph-backed investigation; it does not replace it. Model-reported confidence is not the same as verified correctness, and the model should be allowed to return an insufficient-evidence assessment when the available context does not justify a strong conclusion.

Keeping these responsibilities separate makes the system easier to reason about: FalkorDB provides connected context, Spring Boot orchestrates the investigation, and the AI model contributes an additional assessment.

A walkthrough: investigating three incident scenarios

To evaluate the prototype, I created three predefined incident scenarios.

Scenario 1: Checkout payment failures

The first scenario involves payment failures, with evidence associated with the payment service and a signal involving the payment database.

The expected leading hypothesis is dependency degradation. The important part is not simply producing that label; it is making the dependency signal available as a reason to investigate the database and its health.

Scenario 2: Checkout failures after a deployment

The second scenario focuses on failures occurring shortly after a checkout deployment.

Here, deployment regression is the expected leading hypothesis. The deployment timing gives the engineer a useful lead, but further investigation would still be required to establish whether the release actually introduced the failure.

Scenario 3: Checkout requests timing out downstream

The third scenario involves checkout requests timing out while evidence points toward the payment service.

In this case, dependency degradation should rank ahead of deployment regression. The scenario helps test whether the investigation can distinguish a downstream dependency signal from a deployment-related explanation.

Together, these scenarios test whether TraceOps can produce different leading hypotheses for different evidence patterns, rather than treating every checkout failure as the same problem.

They are also a reminder that incident investigation is not just classification. The system needs to communicate what it found, why a hypothesis is worth exploring, and what information is still missing.

Evaluating the prototype

I wanted a repeatable way to test the investigation workflow instead of relying only on a successful live demo.

I created a scenario evaluation script that checks the expected leading hypothesis and evidence coverage across the predefined incidents.

The current evaluation reports:

100% top-one hypothesis accuracy across the three predefined scenarios.
100% evidence coverage across those scenarios.
100% precision and 100% recall for the deployment-regression case in this small evaluation set.

These results are encouraging, but they need to be interpreted carefully. Three predefined scenarios are not enough to establish general diagnostic accuracy. The scenarios test whether the current implementation behaves as expected on selected cases; they do not prove that it will correctly diagnose unfamiliar production incidents.

The next evaluation stage would include a larger collection of incidents, ambiguous evidence, conflicting signals, missing telemetry, and cases where insufficient evidence is the correct answer.

For an incident investigation tool, knowing when not to make a strong claim is as important as selecting the right hypothesis.

What I learned while building TraceOps

1. Better context can be more valuable than a more elaborate prompt

My biggest takeaway was that AI reasoning depends on the quality of the context it receives.

An isolated alert gives a model limited information. Connecting that alert to a service dependency, a deployment, and other observed signals provides a more useful starting point for investigation.

This is why the graph is central to TraceOps rather than simply a visualization added to an AI application.

2. Correlation is not causation

One of the easiest mistakes in incident analysis is to treat temporal proximity as proof. If a deployment happens just before an outage, it deserves investigation, but it is not automatically responsible.

Building TraceOps made me think more carefully about how a system should communicate hypotheses. A useful investigator should distinguish observations, supporting signals, alternative explanations, and confirmed causes.

3. Explainability has to be designed into the workflow

Displaying a final answer is not enough. Engineers need to understand what evidence contributed to a hypothesis and what they should check next.

This pushed me toward explicit graph queries, understandable heuristic rules, and structured model responses. These design choices also make the application easier to test and debug.

4. Evaluation should challenge the system, not just demonstrate it

A successful demo can show that an application works on one example. A repeatable evaluation helps reveal whether it behaves consistently across different cases.

The three seeded scenarios gave me a practical starting point, but they also exposed the limits of small test sets. Broader evaluation needs to include cases where the evidence is incomplete, conflicting, or misleading.

5. Building solo taught me to prioritize the investigation loop

Working on the project independently meant making deliberate scope decisions. Rather than attempting to build a complete observability platform, I focused on one core workflow: retrieve the incident context, traverse the graph, rank plausible hypotheses, request an AI assessment, and expose the result through an API and frontend.

That end-to-end loop became the foundation of the prototype. It also gave me a clearer roadmap for what should come next.

What comes next?

TraceOps is a prototype, and there is plenty of room to improve it.

My next priorities are:

Expand the evaluation dataset with more realistic and ambiguous incidents.
Add richer historical incident and verified-resolution data to improve investigation context.
Strengthen evidence provenance so that every hypothesis can be traced to the underlying observations.
Improve confidence calibration and measure how often the system appropriately abstains when evidence is insufficient.
Develop a controlled remediation workflow in which recommended actions require explicit human approval before any production-changing operation.

The last point is particularly important. An incident investigator should not silently turn a plausible hypothesis into an automatic rollback. Recommendations and execution are different responsibilities, and production-changing actions need safeguards.

Final thoughts

TraceOps started with a question: could connected data help AI investigate incidents more responsibly?

Building it with FalkorDB, Spring Boot, and JevModel gave me a practical way to explore that idea. The project combines graph-based context retrieval, explicit hypothesis ranking, and structured AI assessment in a single investigation workflow.

The prototype is still early, and its evaluation results apply only to the predefined scenarios. But the central idea is one I want to keep building on: incident response should not depend on an AI system sounding certain. It should depend on the quality of the evidence, the transparency of the reasoning, and the engineer's ability to verify the next step.

TraceOps is my step toward incident investigation that connects the dots before drawing conclusions.

Built for WeMakeDevs FalkorDB Graph Hacks 2026. By Team MinionMR