ExpenseBot 費用審核智能體的構建教訓：別只信答案，更要查閲日誌

canonical_url:https://valentiuskryptix.com/3-surprising-failures-from-testing-my-ai-auditing-agent/

A chatbot answers questions. An agent is different: it is given a goal and a toolbox, then decides for itself which tool to call, what to pass in, and when to stop. For my “Agentic AI Workflow” milestone during an internship at Valentius Kryptix, I built one, and I set myself a harder requirement than “it works.” I had to prove it.

The project

ExpenseBot reviews employee expense claims against an invented company policy. It is written in Python with LangGraph and has three tools: a policy search, a currency converter with fixed rates, and a calculator. The calculator parses expressions into a syntax tree and allows only numbers and basic operators, instead of using eval, because model output feeding into code should be treated as untrusted input.

A system prompt acts as the agent’s job description: look up the rule, convert foreign currency, do all arithmetic through the calculator, and if no policy matches, say so and refer the user to Finance instead of inventing a rule.

Take three hotel nights at 200 euros each. The agent finds the $250 international cap, converts to $216 per night, sees it is under the cap, calculates 648, and approves the claim. Every number comes from a tool call, so it can be traced in the log.

How I tested it

I wrote 12 test claims with expected results before running anything. A test passed only if the final answer was correct and the agent had used the tools that answer depends on.

Three lessons from failures

1. The bug was in the tools, not the prompt. A taxi claim was told no policy existed. My instinct was to rewrite the prompt, but the log showed my search scoring treated “domestic” as important as “taxi,” so three rules tied and the taxi rule lost. Fixing the search fixed the problem.

2. A right answer can hide a wrong process. A smaller model returned correct totals but did the multiplication itself instead of calling the calculator. I counted those as failures. A system that sometimes guesses right is not one that computes.

3. The plumbing matters too. Rate-limit errors were being recorded as test failures, so I made the script save progress and resume later. GitHub also blocked a push because of a real API key in a template file, a good reminder to rotate any exposed credential, not just hide it.

What the numbers say

The agent scored 10 to 11 out of 12 depending on the run. Those scores shift even at temperature zero, so I treat them as a rough range. Twelve cases are enough to find bugs, not to claim general reliability. The prototype also uses keyword search, invented policy data, and handles one claim at a time, so embedding-based search and multi-expense claims are my next steps.

Takeaway

Read the tool-call log before touching the prompt, test the process as well as the output, and report your results with the failures and limits included.

👉 Read the full article: 3 Surprising Failures from Testing My AI Auditing Agent

Code and test cases: https://github.com/Yogendra404/expense-agent