作者打造 ToolCallBench pip 工具，測試本地大語言模型的工具呼叫能力

ToolCallBench tests tool calling on your own models and hardware. A demo run on a 16GB MacBook, and how to try it on yours.

I wanted a simple answer to a simple question: which local model can I trust to change files on my own machine?

Not which model is smartest. Not which one wins a leaderboard. I have a MacBook M4 with 16GB of memory, I run models through Ollama, and I wanted to know what actually works there. When I looked, I couldn't find a test that ran on my hardware, with my settings, and checked the result the way I would: did the right thing happen to the files?

So I wrote one. It's called ToolCallBench, it's on PyPI, and you can point it at any local model you run: Ollama, or any OpenAI-compatible server like llama.cpp, LM Studio or vLLM. The model just needs to support tool calling.

This post covers what the tool does and shows a demo run. The four models in that run are only the ones I happened to have pulled. They're an example of the output, not a list of the best models.

What it tests

The model gets six tools: read a file, list a folder, search, edit, apply a patch, and run a command. Then it gets 20 small tasks on a tiny sample Python project, called "shelf", which has two tests that fail on purpose.

The tasks come in five groups:

• Read: find a config value, list a directory, count rows in a CSV
• Edit: change a value, bump a version, add a function, apply a diff
• Run: run the tests, run a script and report a number
• Chain: do two or three steps in order, like fix a bug and then confirm with the tests
• Failure: answer without tools when none are needed, or recover when a file doesn't exist

Twelve of the 20 name the tool in the prompt. Eight don't, and the model has to work out what to use.

How a run is scored

This was the part I cared about most. There is no second model grading the first one. A run passes only if every tool call was valid, the right tools were used, and the repository ends in the expected state. If the model says "done" but the file is wrong, that's a fail.

When a run fails, it gets a reason: invalid arguments, wrong tool, a patch that didn't apply, the call limit, or the repo ending up different from what was expected. That's more useful than a single score, because "wrong tool" and "bad patch" need different fixes.

The settings are the same for every model: temperature 0, 16k context, q8_0 KV cache with flash attention, same system prompt. The report records all of it, so a number comes with the conditions that produced it.

A demo run

I ran four models I already had: 20 tasks each, one run per task, Ollama 0.32.15 on a MacBook M4 with 16GB. Swap in your own and you get the same kind of table for them.

What stood out in my demo

Running commands was easy. Editing wasn't. All four models passed all three run tasks. Three of the four passed every read task. If I had only tested reading and running, I would have said all four are fine. The edit group is where they separate: 5 out of 5, 3, 2 and 1.

The fastest model was not the cheapest to run. qwen3.5:0.8b did around 80 tokens a second and finished a task in under 4 seconds. It also failed 4 of 5 edits, and five of its nine failures were the "outcome mismatch" kind: the calls looked fine, but the files ended up wrong. Fast and wrong costs you the time you spend checking.

gemma4:e4b was the most accurate and the slowest per task. 19.5 seconds a task against 7.7 for qwen3:8b. I haven't dug into why it's slower per task even with a higher token rate than the 8B models, so I won't guess in a blog post. If you know, tell me.

llama3.1:8b got zero of four chain tasks. Single steps were fine. Putting two or three together was not. Its failures were mostly wrong arguments and patches that didn't apply.

One task failed for every model. chain-04, "fix both failing tests", failed for all four, including the best one. That's either a genuinely hard task or a problem with my check, and I don't know which yet. It's the first thing I'm going to look at, and I'd rather say that here than hide it.

What the demo doesn't tell you

I want to be straight about this, because a benchmark post that oversells is worse than no post.

• It's one run per task. With 20 tasks, each task is 5 points, so the gap between qwen3:8b and llama3.1:8b is three tasks. That's inside the noise.
• It's one laptop and one small sample project. Your project is bigger and messier than mine.
• The tasks are mine. They reflect what I think matters, and I'm sure I've missed things.

So don't read the table as a ranking of models. It shows what the tool reports. The useful number is the one you get for the model you actually plan to use, on your machine, with your settings.

Try it on your models

pip install toolcallbench
toolcallbench init

init creates a configs/ folder. Put your Ollama model names in configs/my-models.yaml, then:

toolcallbench doctor --probe
toolcallbench run --config configs/my-models.yaml
toolcallbench report

doctor --probe checks that the server is reachable and sends each model a quick test call, so you find problems before a long run instead of after. The report comes out as Markdown and HTML in ./results.

It also works with any OpenAI-compatible local server (llama.cpp, LM Studio, vLLM), and there's a scripted backend that runs the whole pipeline without a real model, which is useful if you want to check your setup first.

The code is MIT licensed: https://github.com/herinpatel15/toolcallbench

What I'd like from you

Run it on the models you use and tell me what you get, especially if a model surprises you. And if you read the task list and think "obviously you need to test X," open an issue. Twenty tasks is a start, not the answer.