LLM 回應緩慢時，GPU 未必是問題所在

Your LLM Is Slow. The GPU Might Not Be the Problem.

A practical approach to tracing queueing, retrieval, prefill, and decoding in production AI systems.

A user asks a question.

Your application authenticates the request, searches a knowledge base, builds a prompt, calls an inference endpoint, and streams an answer.

The user waits.

Your dashboard says:

AI request duration: 11.4 seconds

That number tells you the experience was slow.

It does not tell you what to fix.

Should you change the model? Add replicas? Tune retrieval? Reduce the prompt? Investigate queueing? Fix a buffering proxy?

Before changing infrastructure, I would ask a more basic question:

Where did the time go?

One Request Contains Several Different Systems

Consider a retrieval-augmented application:

Client
│
API gateway
│
Authentication
│
Query embedding
│
Retrieval
│
Reranking
│
Prompt construction
│
Inference endpoint
│
Response streaming
│
Client rendering

This is an illustrative architecture, not a requirement that every application use all these components.

The important point is that the user experiences one operation while the application executes several.

A single request_duration measurement collapses those operations into one number.

That is useful for measuring the symptom. It is insufficient for choosing the intervention.

Google’s SRE guidance makes this distinction explicit: monitoring needs to answer both “what is broken?” and “why?” User-facing symptoms and internal causes are related, but they are not interchangeable. sre

Define the Measurement Boundary First

“Time to first token” sounds precise until two teams measure it differently.

An inference service might start its timer when it receives a request.

Your backend might start when it calls that service.

The browser might start when the user submits the question.

Those are different boundaries.

For an application, I would record at least these timestamps:

t0: Application receives request
t1: Retrieval starts
t2: Retrieval completes
t3: Model request is dispatched
t4: First content chunk arrives from model service
t5: First content chunk is forwarded to client
t6: Final response chunk is forwarded

If client instrumentation is available, add:

t7: First answer content is rendered

Name the measurements according to what they actually observe.

For example:

application_first_content_latency = t5 - t0
model_call_first_content_latency = t4 - t3
retrieval_duration = t2 - t1
stream_forwarding_delay = t5 - t4

A content chunk is not necessarily one model token. Avoid labeling a transport-level measurement as per-token latency unless the underlying data supports that interpretation.

Also, do not subtract timestamps from different machines as though their clocks were perfectly aligned. Prefer local elapsed-time measurements and a tracing system designed to represent distributed execution.

Separate Waiting From Working

Suppose the model call takes 2.3 seconds before returning its first content.

That observation alone does not reveal how much time was spent:

Traveling to the service.

Waiting for admission.

Waiting for execution resources.

Processing uncached prompt tokens.

Producing and returning the initial output.

If you operate the serving engine, expose those distinctions where possible.

If you use a hosted API and do not receive internal timing information, preserve that uncertainty.

Write:

Model-call first-content latency increased.
Internal queueing and execution times are not available.

Not:

Prefill became slower.

The second statement requires evidence that the first does not provide.

Instrument Request Paths, Not Just Components

A database dashboard can tell you the database is slow.

It cannot necessarily tell you whether the specific failed answer used that database path, a fallback path, or a cached result.

That is where distributed tracing becomes useful.

OpenTelemetry’s context-propagation model carries trace context across service boundaries. Logs can include trace and span IDs, making it possible to connect records to the execution that produced them. opentelemetry

For a proposed AI application, I would start with a span tree like this:

POST /answer
├── authenticate
├── embed_query
├── retrieve_documents
├── rerank_documents
├── construct_prompt
└── generate_answer
├── wait_for_first_content
└── consume_response_stream

The inference server may contribute additional internal spans if it is instrumented and propagates context appropriately.

Do not invent visibility that your deployment does not provide.

The goal is a navigable execution path, not an impressive-looking diagram.

Attach the Context That Makes Comparisons Useful

A trace becomes more informative when it records which version and workload it represents.

For the application design above, I would consider attributes such as:

service.version
deployment.environment
model.identifier
prompt.template_version
retrieval.index_version
input.token_count
output.token_count
retrieval.document_count

These are suggested fields; define naming and semantics consistently in your implementation.

The principle is more important than the exact names:

When performance changes, you need to identify what else changed.

Google’s SRE Workbook specifically recommends monitoring binary versions, feature flags, and dynamic configuration versions to help diagnose production regressions. sre

For an AI application, I would extend that discipline to prompt templates, retrieval configuration, and model routing.

Do not put secrets, authorization headers, or raw private prompts into telemetry by default. Decide what is necessary to collect, who can access it, and how long it should be retained.

An Illustrative Incident: Slow Answers After a Retrieval Change

Imagine this before-and-after comparison:

Measurement

Before

After

Retrieval duration

180 ms

190 ms

Documents selected

5

18

Input tokens sent to model

2,400

10,800

Model-call first-content latency

650 ms

2,300 ms

These are invented numbers for an engineering example, not a benchmark.

The database did not become dramatically slower.

The model did not change.

But the request sent to the model became much larger.

That gives us a hypothesis:

A retrieval-configuration change increased prompt size and contributed to higher first-content latency.

It does not yet prove the cause.

The next step is a controlled comparison:

Use the same evaluation questions.

Hold model selection and output limits constant.

Compare the old and new retrieval policies.

Control or record concurrency.

Record cache conditions where they are observable.

Measure answer quality as well as latency.

Reducing retrieved context is not automatically a successful optimization if it also removes the evidence needed to answer correctly.

The experiment needs both performance and quality criteria.

Compare Like With Like

An aggregate latency chart mixes workloads.

A short answer generated from a small prompt is not the same workload as a long answer generated from a large document collection.

Before concluding that a release caused a regression, I would break down results by a manageable set of dimensions:

Model or routing destination.

Input-length range.

Output-length range.

Application version.

Prompt-template version.

Deployment region.

Relevant request category.

Use request-specific details in logs and traces rather than turning every identifier into a metric label.

For example, a request ID is useful for finding one execution. It is not a useful grouping dimension for an application-wide latency dashboard.

The objective is to distinguish a genuine slowdown from a change in traffic mix.

“What Changed?” Is a Better Starting Point Than “What Looks Abnormal?”

When an incident begins after a release, the release is an obvious candidate.

But a temporal match is not proof.

The same caution applies to:

A model update.

A larger prompt.

A cache miss.

A traffic surge.

A dependency warning.

I would keep a short hypothesis table during an investigation:

Hypothesis

Supporting evidence

Evidence still needed

Retrieval change increased prompt-processing cost

Larger inputs after configuration change

Controlled old-versus-new comparison

Queueing increased under load

Higher first-content latency at higher concurrency

Serving-side admission timing

Streaming is buffered downstream

Backend receives content before client does

Proxy and client timing comparison

This makes uncertainty visible.

It also discourages the familiar incident pattern where a plausible explanation becomes the accepted explanation before anyone tests it.

An AI Investigation Assistant Should Have to Show Its Work

An LLM can help summarize traces, compare configurations, and generate candidate explanations.

That is a proposed role—not a claim that it can reliably identify root cause from arbitrary telemetry.

If I added an AI assistant to this workflow, I would require its output to separate:

Observation
Hypothesis
Supporting evidence
Missing evidence
Proposed test
Potential operational risk

A useful response might be:

Input-token counts increased after prompt-template version 12 was deployed. The sampled traces also show higher model-call first-content latency. This is consistent with increased prompt-processing work, but internal queue timing is unavailable. Compare versions 11 and 12 under matched load before attributing the regression to prefill.

That is more actionable than:

Root cause found: prompt too large. Roll back immediately.

The first answer explains what is known and proposes a test.

The second turns a hypothesis into an operational instruction.

A Minimal Investigation Checklist

Before buying more hardware or replacing the serving engine, I would check:

Does the latency measurement represent the user’s experience or only one backend call?

Can a failed request be followed across retrieval, generation, and streaming?

Are queueing and execution distinguishable where the infrastructure exposes them?

Did prompt size, output length, or traffic mix change?

Are deployments and configuration changes recorded?

Does the proposed fix preserve answer quality?

Can the suspected cause be tested without changing several variables at once?

This does not require collecting everything.

It requires collecting enough connected evidence to answer the next diagnostic question.

Where This Connects to My Book

I’m Ahmed Adawy, the author of AI Systems Engineering: From Prototype to Production.

The book sample explores the inference lifecycle, model-weight and KV-cache budgeting, prefill and decode, and serving metrics. It also includes an educational asynchronous serving simulator. ppl-ai-file-upload.s3.amazonaws

That foundation helps frame a useful distinction:

Understanding how an inference engine works is necessary for diagnosing its behavior—but application latency also includes the systems around it.

The simulator is an explanation tool, not a production benchmark. Likewise, the example in this article is a diagnostic scenario, not a claim of measured results from a customer deployment.

If you want to examine the book’s technical level and approach, start with the free sample:

Explore the book and read the sample on Leanpub

Disclosure: this is my book, and I earn from purchases.

The Question to Ask Before “Do We Need a Bigger GPU?”

A slow answer is an observation.

“Insufficient GPU capacity” is one possible explanation.

The engineering work is connecting the two with evidence.

Before changing hardware, establish where the request waited, what it processed, which dependencies it used, and what changed.

Otherwise, you may improve a component that was never the bootelneck.