跳到正文
DEV Community· Dhanush Kumar Sivaji·· 3 小時前AI 評分63

AI FDE 之路 #01：LLM 應用請求流程剖析

The path to AI FDE #01: Anatomy of an LLM app request

AI 導讀

Dhanush Kumar Sivaji 將一次 LLM 應用請求拆解為九個步驟，涵蓋用戶端呼叫、輸入驗證、提示詞組裝、模型生成、串流回傳、輸出驗證、重試與記錄。

正文

I've spent 8+ years building React/TypeScript front ends. Now I'm learning applied AI in public, working toward AI Engineer, then AI Solution Architect, and (I hope) AI Forward Deployed Engineer. Before tokens, RAG or agents, I wanted one map: what actually happens when a user clicks "Summarize"?

Anatomy of an LLM app request: a nine-step flow from client to logging, with steps 5 and 6 (prefill and decode) inside the model provider

Each step below gets its own figure first, then the short version in words.

Before the call: steps 1–4

Where each piece runs: the browser posts to your FastAPI backend, which holds the API key and calls the model provider; inside the backend a Pydantic model either lets a valid body through or FastAPI returns a 422 before any tokens are spent

1. The client sends a request. From the browser's side, an LLM feature is just an HTTP call to your backend: POST /summarize with the user's text, plus a conversation id if it's a chat. OpenAI's docs say never to expose an API key in client-side code, so the key lives on your server and only the server talks to the provider.

2. Your API validates the input. FastAPI checks the body against a Pydantic model. Invalid data never reaches your handler: FastAPI raises a RequestValidationError, and its default handler returns HTTP 422. Once validation passes, Pydantic guarantees the fields match their declared types. It's the cheapest place to stop bad input.

Each request stands alone: three successive requests resend the system prompt and the whole history, so input grows each turn; optional server-side state still bills earlier input tokens

3. You assemble the prompt. The model sees only what you send. Anthropic's Messages API is stateless, so you resend the full history every time. OpenAI also calls each request independent and stateless, and offers optional server-side conversation state, but notes that earlier input tokens in the chain are still billed on each turn. Later in this series, retrieved documents (RAG) join the prompt here.

Tokens and the context window:

4. Count tokens and check the budget. Models read tokens, not characters, and tokenizers are tied to the model, so counts differ between models. System prompt, tool definitions, history and the output all share the context window. If the input alone is over the limit, the request fails (Anthropic returns a 400, "prompt is too long"). Estimate tokens before sending and set a maximum output length.

Inside the model: steps 5–6

Prefill vs decode: a timeline from request sent through network and queue, a prefill block, the first token, then one decode step per token until a stop condition; TTFT and time per output token are marked

5. Prefill: the model reads all the input at once. Inference has two phases. In prefill, the model processes every input token in parallel. It's compute-heavy, and it's the main model-side driver of time to first token (TTFT). Network time and provider queueing add to that pause.

6. Decode: one token at a time. Then the model generates autoregressively: score every token in the vocabulary, pick one, append it, repeat. A KV cache keeps earlier work so each step doesn't start over, and the per-step pace is the time per output token. It stops at an end-of-sequence token, a stop sequence or the token limit.

How the next token gets picked: a toy probability distribution for

How it picks is a sampling setting: greedy, or sampled with temperature, top-k or top-p. Don't assume those settings exist on every model: Anthropic's docs say Claude 4.7 and later return a 400 if you set them to non-default values.

After the call: steps 7–9

Streaming with server-sent events: a sequence diagram where the provider sends token deltas to your server, which relays them to the browser; the stream ends with a final event or with an error event after a 200

7. Stream the tokens back. OpenAI and Anthropic both stream tokens as server-sent events (SSE) while decoding is still running, so steps 6 and 7 overlap. Two catches from the docs: an error can arrive mid-stream, after a 200 status, and partial output is harder to moderate than a finished response.

Validating the model's output: a decision flow that checks for a refusal, a response cut off at the token limit, and whether the output parses into the Pydantic model, before FastAPI's response_model check returns typed JSON

8. Validate the output too. Structured outputs constrain the model to a JSON Schema you supply (OpenAI supports a subset), but OpenAI's guide says they can still contain mistakes, and documents two cases to handle: refusals and responses cut off at the token limit. If you stream, validate once the stream completes. FastAPI's response_model is a second check: a mismatch becomes a server error, not wrong data. A valid shape isn't a correct answer; evals cover correctness later in this series.

Retry or not: a decision tree that sends rate limits, overloads and connection errors to limited retries, quota, billing and spend-cap errors to no retry, and bad requests or credentials to fix-first

9. Retry safely, then record. The same status can mean different things: a 429 can be a rate limit, but Anthropic also uses it when you hit your monthly spend cap, so read the error body before retrying. Never retry quota or billing errors. For transient errors, honor Retry-After when present, otherwise use exponential backoff with jitter, and cap attempts and total time. Official SDKs already retry some errors (Anthropic's twice by default), so count that in your limits rather than nesting loops. Then log input/output token usage, latency and the provider's request id. Those records feed debugging, cost tracking and, eventually, evaluation.

Takeaway

As a front-end engineer, I found this oddly familiar: it's a request lifecycle with an unusual backend in the middle. Two of the nine steps are the model. The other seven are contracts, budgets, streams and failure handling, the parts an engineer owns. My working rule from here on: treat the model as an unreliable, metered, remote dependency, and put validation on both sides of it.

Where this fits in my roadmap: this post is the map I'm following into the foundations block: Python typing and async, FastAPI and Pydantic, and LLM mechanics (tokens, context, inference). Next post: Python for AI backends: a TypeScript engineer's refresher.

Over to you: which of these nine steps broke first in the first LLM feature you built?

Sources

  1. OpenAI API reference: Authentication (don't expose API keys in client-side code). https://developers.openai.com/api/reference/overview
  2. FastAPI: Handling Errors (RequestValidationError and its default handler). https://fastapi.tiangolo.com/tutorial/handling-errors/
  3. FastAPI source: exception_handlers.py (default validation handler returns 422). https://github.com/fastapi/fastapi/blob/master/fastapi/exception_handlers.py
  4. Pydantic: Models (post-validation type guarantees). https://pydantic.dev/docs/validation/latest/concepts/models/
  5. Anthropic: Working with the Messages API (stateless, resend full history; sampling parameters on Claude 4.7+). https://platform.claude.com/docs/en/build-with-claude/working-with-messages
  6. OpenAI: Conversation state (requests independent and stateless; optional state management; earlier input tokens still billed). https://developers.openai.com/api/docs/guides/conversation-state
  7. Hugging Face LLM Course, Ch. 2: Tokenizers (subword tokenization, "token" + "ization"). https://huggingface.co/learn/llm-course/chapter2/4
  8. Anthropic: Context windows (what counts toward the window; oversize input returns 400; token counting). https://platform.claude.com/docs/en/build-with-claude/context-windows
  9. Hugging Face LLM Course, Ch. 1: Deep dive into text generation inference (prefill/decode, TTFT, TPOT, KV cache, sampling, stop conditions). https://huggingface.co/learn/llm-course/chapter1/8
  10. Vaswani et al., "Attention Is All You Need" (2017), the transformer/attention paper. https://arxiv.org/abs/1706.03762
  11. OpenAI: Streaming API responses (SSE, moderation risk for partial output). https://developers.openai.com/api/docs/guides/streaming-responses
  12. Anthropic: Streaming Messages (SSE). https://platform.claude.com/docs/en/build-with-claude/streaming
  13. OpenAI: Structured model outputs (schema adherence, supported subset, refusals, incomplete responses, mistakes still possible). https://developers.openai.com/api/docs/guides/structured-outputs
  14. FastAPI: Response Model (server error instead of invalid returned data). https://fastapi.tiangolo.com/tutorial/response-model/
  15. Anthropic: Errors (429 incl. spend caps, 402 billing, 400 spend limit, 529 overloaded, SDK retries, mid-stream errors after a 200, request ids). https://platform.claude.com/docs/en/api/errors
  16. OpenAI: Rate limits (429/503, Retry-After, backoff with jitter, cap retries, nested SDK retries, don't retry quota/billing errors). https://developers.openai.com/api/docs/guides/rate-limits

Sources checked Oct 18, 2026. Provider APIs change quickly. Re-check model-specific details (status codes, sampling support) right before publishing.

Originally published on The Path to AI FDE on Substack. Subscribe for one diagram-led post every weekday: https://dhanushkumarsivaji.substack.com/p/the-path-to-ai-fde-01-anatomy-of

來源：DEV Community · dev.to

#教學/實踐#部署/工程#推理#OpenAI#Anthropic