AI 應用設有請求速率限制，Agent 仍可能推高帳單

A user clicks “Research this topic.”

Your app accepts one request.

Behind the scenes, the agent searches the web, reads pages, calls the model, retries a failed tool, and searches again.

Your rate limiter sees one request.

Your infrastructure sees a growing chain of work.

Limiting requests at the entrance doesn’t limit everything that happens afterward.

If you’re building an AI feature, here’s how to keep that work under control.

1. Count the work behind the button

Imagine a research task that uses:

• One model call to plan.
• Five searches.
• Ten page fetches.
• Several model calls to analyze results.
• One final call to write the answer.

That’s an illustrative workflow, but it shows the problem: one user action can trigger many billable operations.

Track usage for the entire task, including retries and background work.

A useful starting point is to record:

task_id
account_id
model_calls
tool_calls
input_tokens
output_tokens
elapsed_time
estimated_cost
stop_reason

Keep sensitive document content and credentials out of routine logs.

2. Give every task a budget

A task needs an explicit stopping point.

For example:

const limits = {
  maxModelCalls: 8,
  maxToolCalls: 15,
  maxDurationMs: 90_000,
  maxEstimatedCostUsd: 0.30,
};

These numbers are examples, not recommended defaults.

Choose limits based on the feature, its pricing, and the quality users need.

Enforce them in your application. A prompt saying “don’t spend too much” cannot replace a budget check.

Before starting another operation, check whether the task has enough remaining budget.

3. Remember that cost estimates have uncertainty

You may not know the final charge before a model call completes.

Reserve a conservative allowance before dispatching it, using input size, output limits, and applicable pricing. Reconcile that reservation against reported usage afterward.

Include paid tools, search services, and other billable operations where relevant.

If several workers share a budget, reserve usage atomically. Otherwise, they can all see the same remaining balance and spend it simultaneously.

A budget counter only works if concurrent work cannot bypass it.

4. Bound retries and agent loops

A failed operation can become an expensive loop:

Search failed → retry → model chooses another search → retry again.

Set limits at two levels:

• Attempts for an individual operation.
• Total work for the whole task.

Also detect repeated actions that produce no progress.

If an agent keeps fetching the same page or repeating the same unsuccessful query, stop or ask the user for clarification.

Activity is not evidence of progress.

5. Limit input and output sizes

A request count treats a short question and a huge document as one request each.

Their resource requirements can differ considerably.

Set appropriate limits on:

• Upload size.
• Extracted document length.
• Retrieved pages.
• Tool response size.
• Model output length.

OWASP identifies uncontrolled LLM resource consumption as a risk and recommends controls including input validation, rate limits, and user quotas.

Make limits understandable to users. “This document exceeds the supported length” is more useful than a mysterious failure after a long wait.

6. Make cancellation reach the workers

A user closes the tab.

Does the research stop?

If your backend job continues, the interface disappearing has changed nothing about the remaining work.

Propagate cancellation to queued jobs and workers. Check cancellation before each new model or tool call.

Attempt to cancel in-flight operations where supported, while recognizing that work already performed may still be billable.

Cancellation should stop future work as promptly as possible.

7. Use account limits as well as task limits

A bounded task can still become expensive when someone starts hundreds of them.

Combine:

• Per-task budgets.
• Per-account quotas.
• Concurrent task limits.
• Application-wide spending controls.

Use provider controls as another layer where available. Verify whether a provider’s “budget” actually blocks spending or merely sends an alert.

An alert tells you something happened. A blocking control prevents additional work.

8. Stop with an honest result

When a task reaches its limit, don’t pretend the research is complete.

Return the useful work already collected:

“I reviewed six sources and reached this task’s research limit. Here are the findings so far, along with the questions that remain.”

You can offer an explicit continuation if the product supports it.

A partial answer with clear limits is easier to trust than a confident answer hiding unfinished work.

Before you launch

Test these situations:

• A tool fails repeatedly.
• The agent repeats the same action.
• Several tasks start simultaneously.
• A large document is uploaded.
• The user cancels halfway through.
• An account reaches its quota.

Check whether your workers actually stop—not just whether the interface displays an error.

One question worth asking

Before adding another autonomous step, ask:

“What is the maximum work one user action can trigger?”

If you can’t answer, measure the workflow and add enforceable limits.

A useful agent needs room to work. Your application needs control over how much work it can start.

Does your AI feature limit incoming requests—or the full task behind each request?