三名 AI 調查員與一個程式碼庫：AI Agent 背後的隱藏問題

Thanks for support..
Release: "Sentinel-IR-MCP v0.2.2 on GitLab" (https://gitlab.com/jakubrojicek11/sentinel-ir-mcp/-/tree/v0.2.2?ref_type=tags)

Package: "sentinel-ir-mcp@0.2.2 on npm" (https://www.npmjs.com/package/sentinel-ir-mcp/v/0.2.2)

The 30-second summary

Imagine asking an AI to investigate a software project. Should it read every line of code, receive a short summary, or start with a summary and open the original code only when it needs more evidence?

I tested these three approaches using Sentinel's live benchmark with GPT-6 Astra. Across 87 questions, the full-code approach answered 84 correctly, the compact-information approach answered 82 and left five unresolved, while the hybrid approach answered all 87 correctly.

The hybrid approach also used approximately 71% fewer estimated input tokens than the full-code approach. This is a promising result from one controlled test, not proof that AI agents are now error-free.

"Read the original benchmark and inspect the data yourself." (https://dev.to/jackymencz/sentinel-benchmark-livejson-ifl)

Imagine hiring an investigator who has to read an entire library

Imagine your company hires an investigator to inspect a software project before it goes into production.

The investigator has a straightforward assignment:

Find out what the program does, what it can access, whether it communicates over a network, whether it writes files, and whether it contains operations that deserve extra attention.

There is one complication.

The project contains hundreds or thousands of lines of code. The investigator cannot simply look at the name of the project and know what happens inside it. They need evidence.

Now imagine giving three versions of the same investigator the same assignment.

The first receives every original document.

The second receives a structured report prepared in advance.

The third starts with the report but can return to the original documents when the report does not contain enough information.

Which one will do the best job?

That is the central question behind my Sentinel benchmark.

There is an important distinction here: these are not three independent AI agents arguing with one another or exchanging messages. The benchmark compares three ways of supplying information to the same named model, GPT-6 Astra.

The difference is not simply how intelligent the investigator is. It is how much relevant evidence the investigator receives, and how the system responds when that evidence is insufficient.

Investigator number one: read everything

The first approach is called RAW.

The model receives the original source code and answers questions about it.

Think of a detective who receives the complete case file, including witness statements, photographs, reports, and every page of supporting evidence.

Nothing needs to be summarized before the detective begins.

This has an obvious advantage. The relevant detail might be buried in an unexpected place, and the original material is available for inspection.

But there is a cost.

Every time the model needs to investigate a question, the supplied information can contain much more material than that question actually requires.

Suppose you ask whether a program exposes an HTTP endpoint. The answer might be visible in a handful of lines. Yet the model may also receive code responsible for unrelated calculations, formatting, logging, and other operations.

The extra material is not necessarily harmful. It is simply not always relevant to the current question.

In AI systems, the pieces of information supplied to a model are often represented as tokens. Tokens are the units models process when reading text and producing responses.

A larger input can require more processing and may increase usage costs, depending on the model and service.

The important point is that giving an AI more information is not automatically the same as giving it better information.

Sometimes, the extra pages contain exactly what the investigator needs. Sometimes, most of them are irrelevant to the question.

Investigator number two: start with a structured map

The second approach is called IR, short for Intermediate Representation.

Instead of receiving only the original code, the model receives a structured description of important facts extracted from that code.

Imagine receiving a building's floor plan rather than having to inspect every room before answering a basic question.

The plan might identify the entrances, exits, electrical rooms, and locations of restricted areas. You can answer many questions from that map without exploring the entire building.

Sentinel-IR applies a similar idea to software.

Its output can identify facts such as:

Which functions the program exposes to other parts of the system.
Which HTTP routes are present.
Which external libraries are imported.
Which environment variables the program reads.
Which network, file-system, or process-related operations appear in the code.
Which security-relevant patterns deserve further inspection.

This is more than asking an AI to write a shorter version of a long file.

A normal summary might say, "This program runs a server and processes requests."

A structured representation can instead record individual routes, imported modules, operations, and other specific facts in a predictable format.

The distinction matters because the AI is no longer expected to rediscover every structural detail from the original text whenever it needs to answer a question.

It receives a prepared map.

There is a catch, however.

A map is only useful if it accurately records the important features of the territory.

If a relevant operation is missing from the representation, the model may never see the evidence it needs. And if the representation simplifies something too aggressively, the model might reach the wrong conclusion.

A compact representation is therefore not automatically a better representation.

It must preserve the information that matters.

The surprise: a summary can be bigger than the original

This is where the story becomes interesting.

People often assume that a compressed or summarized version of something must always be smaller than the original.

That is not true for structured software information.

In one small example in the published benchmark, a 27-line HTTP server had an estimated input size of 163 tokens for the original source and 272 tokens for its IR representation.

The structured version was larger.

Why?

Because the original file was already relatively small. A structured report also needs labels, field names, lists, and descriptions to make its facts understandable to the model.

Those additional elements can cost more than the original text when the source is short.

Now consider a much larger file.

For a sample of approximately 393 lines, the benchmark records an estimated 2,861 tokens for the source and 734 for the IR representation.

For a sample of approximately 951 lines, the figures are 7,186 tokens for the source and 498 for IR.

Suddenly, the map becomes much smaller than the territory.

That is the practical reason to investigate this approach.

It is not a promise that every file will shrink. It is an attempt to avoid repeatedly sending large amounts of source code when a compact set of relevant facts can answer the question.

The larger the difference between the original material and the useful information required for a task, the more opportunity there may be to save input tokens.

Investigator number three: a map, with access to the original evidence

The third approach is called IR + RAW.

It combines the two previous approaches.

The model starts with structured information. When that information is not sufficient, the workflow can escalate and consult the original source.

Return to our building inspector.

The inspector begins with a floor plan. It shows the main entrances, exits, and restricted areas.

Most questions are easy to answer from the plan.

But then the inspector notices a door leading into a room marked as restricted. The plan identifies the door, but it does not explain what happens inside the room.

A sensible inspector does not guess.

They request access to the original evidence.

That is the idea behind the hybrid approach: use compact information where it is enough, and retain a path back to the original source when more detail is required.

This is different from choosing between two extremes.

The system does not have to send everything every time, and it does not have to pretend that its initial summary contains every possible answer.

In the benchmark, the IR + RAW variant recorded 93 model calls for 87 questions. That means six additional calls beyond the one-call-per-question baseline, consistent with targeted escalation in this test.

This additional work matters. Access to the original source is not free, and extra calls can add time.

But neither is a wrong answer free. In a software security workflow, an incorrect conclusion about a dangerous operation can be far more expensive than an additional investigation.

The real design question is therefore not, "How can we eliminate every extra call?"

It is, "When is the extra call worth making?"

What did the actual benchmark show?

The published Sentinel benchmark records a live run using GPT-6 Astra across 87 questions.

Here are the reported results.

Approach| Correct answers| Unresolved| Estimated input tokens
RAW| 84 of 87| 0| 279,476
IR| 82 of 87| 5| 58,549
IR + RAW| 87 of 87| 0| 80,340

The accuracy figures reported by the benchmark are 96.6% for RAW, 94.3% for IR, and 100% for IR + RAW.

The differences tell us something important.

First, the compact representation substantially reduced the measured input volume.

IR used approximately 79% fewer estimated input tokens than RAW.

Second, reduced input volume did not automatically preserve the full result.

The standalone IR approach left five questions unresolved. It used less input, but it did not answer every question.

Third, the hybrid approach recovered the missing answers in this particular run.

IR + RAW used approximately 71% fewer estimated input tokens than RAW and answered all 87 questions correctly according to the benchmark's expected answers.

The results support further investigation into a system that combines compact facts with access to original evidence.

They do not establish that the hybrid approach will always outperform the alternatives, or that it is ready to make unsupervised production decisions.

Eighty-seven questions are useful evidence, not a guarantee about every possible codebase.

One important detail about the numbers

There is a technical limitation that matters when interpreting these results.

The benchmark estimates token counts using the length of the text divided by four. It applies that approximation consistently across the variants.

These figures are therefore useful for comparing the approximate size of their inputs. They are not exact measurements of the model's actual tokenization or the money charged by an AI provider.

The distinction matters because real token counts vary with the content and tokenizer, and costs also depend on the model and the service's pricing.

The elapsed times in this run reinforce another point: fewer estimated input tokens did not automatically make the hybrid workflow the fastest.

The recorded elapsed times were approximately 155 seconds for RAW, 147 seconds for IR, and 159 seconds for IR + RAW. These are measurements from this particular run, not general performance guarantees.

So the strongest result here is not that the hybrid system is always cheaper, faster, or more accurate.

It is that this hybrid configuration answered all the tested questions correctly while using substantially fewer estimated input tokens than the full-source approach.

That is a specific, testable claim.

It is also a much more useful claim than simply announcing that AI has become smarter.

Why this is a security problem, not just a question of cost

Now imagine applying the same system to a real software project.

A program might read an API key, communicate over a network, write files, or launch another process. These operations can be legitimate. Their presence alone does not prove that a program is malicious.

But they may matter when assessing what the program can do and what could go wrong.

An AI system investigating the project needs to distinguish between several things:

What does the code contain?

What does the available evidence allow us to conclude?

What information is still missing?

And when should the system refuse to make a confident claim?

These are separate questions.

A report identifying a process-launching operation is evidence about the program's structure. It is not, by itself, a complete security verdict.

Likewise, a missing finding does not prove that an operation is absent unless the analysis method is reliable enough to support that conclusion.

This is why compact representations need more than clever compression. They need accurate extraction, carefully defined facts, and a way to handle missing information.

It is also why evaluation matters.

A result that looks impressive in a demonstration must be tested against known answers, difficult cases, and situations where the system should admit that it does not know enough.

The benchmark is a small example of testing this design question with concrete questions and recorded answers rather than relying only on a persuasive demonstration.

What does this mean for the future of AI agents?

The bigger idea extends beyond source code.

AI agents increasingly work with documents, databases, websites, tools, and information accumulated over multiple steps.

In all these settings, the system must decide what information to keep available, what to retrieve, and what to do when the evidence is incomplete.

Imagine a customer-support agent investigating a delayed order. It may not need the customer's entire history. It needs the order number, shipping status, relevant updates, and perhaps access to the original records when a discrepancy appears.

Imagine a research agent. It may start with notes and extracted findings, then reopen the underlying paper to verify an important detail.

Imagine an AI coding assistant. It may need a compact map of a repository before making a change, but it should still inspect the actual implementation before modifying a critical function.

These are examples of the same design principle, not results demonstrated by this particular benchmark.

There is also a broader lesson about multiple agents.

Adding more AI agents does not automatically make a system better. Agents need useful information, clear responsibilities, reliable communication, and a way to detect mistakes. Otherwise, a larger system can simply create more opportunities for confusion.

Sometimes the answer is to add another agent. Sometimes it is to give the existing agent better evidence. Sometimes it is to use a deterministic tool instead of asking a model to infer something that can be checked directly.

The right design depends on the task.

For Sentinel, the question is narrower: can a structured representation provide enough information for many software-inspection questions, while preserving a route back to the original code when it does not?

The live benchmark provides an encouraging result for that approach. The next step is to repeat the experiment across more projects, more questions, different kinds of code, and independent runs, while measuring actual token usage and costs where possible.

That is how a promising result becomes stronger engineering evidence.

The real lesson: don't make the AI choose between ignorance and everything

Let's return to our investigator.

Giving the investigator every document can work, but it may be wasteful.

Giving them only a summary can be efficient, but it may leave important questions unanswered.

Giving them a structured map and the ability to inspect original evidence offers another possibility.

The investigator can use the map for ordinary questions and return to the evidence when necessary.

That is the idea behind Sentinel's IR + RAW approach.

The goal is not to make an AI appear clever by giving it less information. The goal is to give it the right information for the task, recognize when that information is insufficient, and make further investigation possible.

The 87-question benchmark does not prove that this approach is universally superior. It does show why the approach deserves further testing.

And it raises a question that matters well beyond software security:

When we build AI agents, should we focus only on making them think better, or should we also improve how they receive evidence, recognize uncertainty, and decide when to investigate further?

I believe the second part deserves far more attention.

Because an intelligent investigator with incomplete evidence can still reach the wrong conclusion. A well-designed investigative process gives that investigator a chance to discover what is missing.

Ten articles and research papers worth reading next

The Sentinel benchmark explores one specific problem: how an AI system uses information. These ten resources examine related challenges, from context selection and multi-agent coordination to security and evaluation.

"Effective Context Engineering for AI Agents — Anthropic" (https://www.anthropic.com/research/effective-context-engineering-for-ai-agents)
Why the information supplied to a model needs to be managed throughout an agent's work, not just written into a good initial prompt.
"Towards a Science of Scaling Agent Systems — Google Research" (https://research.google/blog/towards-a-science-of-scaling-agent-systems-when-and-why-agent-systems-work/)
Research on when multiple agents improve performance and when coordination can make a system worse.
"A Practical Guide to Building AI Agents — OpenAI" (https://openai.com/business/guides-and-resources/a-practical-guide-to-building-ai-agents/)
A practical introduction to agent architecture, tools, workflows, and safeguards.
"ContextBench: A Benchmark for Context Retrieval in Coding Agents" (https://arxiv.org/abs/2602.05892)
Research examining whether coding agents retrieve and use the relevant parts of a codebase when solving problems.
"SWE Context Bench: A Benchmark for Context Learning in Coding" (https://arxiv.org/abs/2602.08316)
Research on reusing previous experience and summaries across related coding tasks, including the risks of retrieving the wrong information.
"AgentBench: Evaluating LLMs as Agents" (https://proceedings.iclr.cc/paper_files/paper/2024/hash/e9df36b21ff4ee211a8b71ee8b7e9f57-Abstract-Conference.html)
A benchmark evaluating how language models reason, follow instructions, and act across interactive environments.
"Systematic Debugging for AI Agents: Introducing AgentRx — Microsoft Research" (https://www.microsoft.com/en-us/research/blog/systematic-debugging-for-ai-agents-introducing-the-agentrx-framework/)
How to identify where a long AI-agent workflow failed instead of looking only at its final answer.
"Know Your Limits: A Survey of Abstention in Large Language Models" (https://aclanthology.org/2025.tacl-1.26/)
Research on when an AI should refrain from answering because it cannot provide a sufficiently reliable response.
"OWASP Top 10 for Agentic Applications 2026" (https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/)
Security risks associated with AI systems that can plan, use tools, and take actions.
"Evaluate Agent Workflows — OpenAI" (https://developers.openai.com/api/docs/guides/agent-evals)
Practical guidance on evaluating agent runs, examining failures, and testing whether changes actually improve a workflow.

The numbers in this article come from the published Sentinel live benchmark. The interpretation of the results is deliberately limited to that experiment; broader performance, security, and cost claims require further testing.

Original benchmark: https://dev.to/jackymencz/sentinel-benchmark-livejson-ifl