tokenecon 如何為 LiteLLM 和 LangChain 應用核算智能體執行成本並生成收據

Your LLM bill arrives as an aggregate: a provider invoice, a dashboard with a big number. Nobody can answer the questions that actually matter: what did that task cost? What does this run cost per correct answer? A single model call has a predictable price. A looping agent — tool calls, observations, retries, retrieved documents, each another round-trip through a priced model — does not. Most teams discover the total from the cloud bill instead of the architecture.

That's why I built tokenecon, an open-source Python package (MIT, on GitHub) that decomposes agent spend per task and produces receipts: tier used, tokens, cost, and confidence per step. Until now the only way to measure a live agent was the Strands Agents integration. With tokenecon 0.2.2, two more trackers join it — LiteLLM and LangChain / LangGraph — and this is the hands-on walkthrough.

Install it:

pip install tokenecon

tokenecon itself has no dependencies. The integrations are optional: install litellm or langchain-core only for the framework you're tracking.

LiteLLM: two lines

The LiteLLM integration is a CustomLogger that prices every successful call with tokenecon tiers, using the provider usage LiteLLM already attaches to each response — no estimation, no token-counting heuristics:

import litellm
from tokenecon.integrations.litellm import CostTracker

tracker = CostTracker(tier="medium")
litellm.callbacks = [tracker]

litellm.completion(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize our Q3 cloud spend."}],
)

receipt = tracker.receipt("Summarize our Q3 cloud spend.")
print(receipt.pretty())
# request    : Summarize our Q3 cloud spend.
# difficulty : 0.162
# outcome    : ok
#   step 1: tier=medium in=1840 out=312 cost=$0.003088 conf=1.00 litellm call 1 (gpt-4o-mini)
# total_cost : $0.003088

Every successful call becomes one receipt step, in call order, with confidence 1.0 — with provider-reported usage there is no uncertainty in the cost figure. Both the sync and async LiteLLM hooks are covered, and the tracker is thread-safe.

A few useful properties when you wire it into a real app:

• tracker.steps is a snapshot of the priced calls so far, and tracker.total_cost is the running USD total — enough to drive a live spend guardrail.
• tracker.reset() forgets the recorded calls, e.g. between tasks or between users, so one long-lived tracker can serve a whole service.
• Your app probably mixes models. Pass model_tiers to price specific ones differently — keys match the model name exactly or the part after the last /, so "gpt-4o" also covers "openai/gpt-4o":

tracker = CostTracker(
    tier="small",
    model_tiers={"gpt-4o": "large", "claude-opus-4-1": "large"},
)

If a call arrives with no usage data, it records a zero-token step marked "no usage reported" rather than silently dropping it. And the module imports even without LiteLLM installed — instantiating the tracker raises a clear ImportError only when you actually try to use it.

LangChain and LangGraph: one callback

The LangChain integration is a BaseCallbackHandler that prices every completed model call the same way, and it attaches wherever LangChain accepts callbacks — a model, a chain, an agent, or a compiled LangGraph graph:

from langchain_openai import ChatOpenAI
from tokenecon.integrations.langchain import CostTracker

tracker = CostTracker(tier="medium")
llm = ChatOpenAI(model="gpt-4o-mini", callbacks=[tracker])

llm.invoke("Summarize our Q3 cloud spend.")
receipt = tracker.receipt("Summarize our Q3 cloud spend.")
print(receipt.pretty())
# request    : Summarize our Q3 cloud spend.
# difficulty : 0.162
# outcome    : ok
#   step 1: tier=medium in=1840 out=312 cost=$0.003088 conf=1.00 langchain call 1 (gpt-4o-mini)
# total_cost : $0.003088

For a LangGraph graph, attach it the same way in the invoke config: graph.invoke(state, config={"callbacks": [tracker]}).

Both model shapes are handled through one hook: classic LLMs report usage in llm_output["token_usage"], chat models on each message's usage_metadata, and multi-generation runs are summed into the single step for the run. One design decision worth knowing: the handler never raises into your chain (raise_error = False), so a run whose usage can't be found becomes a zero-token step instead of breaking your application. A cost observer should never break the thing it observes.

Bring your own prices

The default tiers (small, medium, large) are illustrative placeholders. For real numbers, pass a tokenecon.ModelTier with your provider's actual price list as the tier or inside model_tiers. Once you do, receipts stop being estimates and become your audit trail.

Why receipts matter: the measured context

Receipts are the unit of cost observability. Accumulate them and they become the dataset your routing policy improves from — and they feed public work that gives the numbers weight:

• Using tokenecon, the same ~3.5M-token agent workload costs $1.37 on one model and $38.72 on another — roughly a 28× spread — while the framework around the model adds at most ~3.9×. Model choice, not framework choice, is the line item.
• The Cost Per Correct benchmark (dataset DOI 10.5281/zenodo.23439614) measures agents in the unit that matters: cost per correct answer, not cost per call. The cheapest model that fails is the most expensive line item.

And when you have receipts of your own, make them shareable. tokenecon ships a badge generator (new in 0.2.2):

python -m tokenecon.badge results.json --format svg -o badge.svg

It takes either a summary JSON ({"model": ..., "tasks_solved": 5, "tasks_total": 5, "total_cost_usd": ...}) or a list of tokenecon receipts, and renders a shields-style badge — or endpoint JSON for a live shields.io badge — so your README can wear its cost per correct the way it wears its CI status. (For graded benchmark runs, the summary form is the honest input: receipts carry no grading information.)

Every agent run should come with a receipt. Start with two lines of code and see what yours actually costs.

Links: PyPI: tokenecon · GitHub · Cost Per Correct leaderboard · Dataset DOI: 10.5281/zenodo.23439614 · Software DOI: 10.5281/zenodo.23427025