如何在正式環境調用 LLM API：JSON、逾時與成本上限

An LLM API integration that works in a local prototype will fail in production the first time a model endpoint stalls for 45 seconds and exhausts your PHP-FPM or Node worker pool, returns markdown code fences around invalid JSON, or burns $400 overnight on an unthrottled retry loop. Shipping reliable AI features inside a production web application requires five engineering guardrails around every model call: strict JSON schema enforcement, dual connect/read timeouts, jittered backoff with model fallback, deterministic prompt caching, and per-tenant token budget caps. Here is how we architect that layer in production.

Why Prototype LLM Calls Break Production Web Servers

Standard REST APIs respond in 30 to 150 milliseconds with deterministic schemas. Large Language Model APIs behave completely differently:

1. Long and Unpredictable Tail Latency: A prompt that averages 1.8 seconds at p50 regularly spikes to 18–35 seconds at p99 during provider load surges. If you call an LLM synchronously inside a web controller, twenty slow requests can tie up every web worker on your server.
2. Non-Deterministic Output Formatting: Asking a model in a system prompt to "respond only with valid JSON" still fails 1–3% of the time when the model wraps the JSON in json ... fences, omits a required key, or truncates mid-object because max_tokens was set too low.
3. Asymmetric Token Cost Explosions: A single bug in a client-side retry loop or an uncapped user input field (pasting a 60-page log file into a chat box) multiplies input token bills immediately.

When we build AI-assisted workflows for clients through our custom web application development services at Loop Origin, we never allow controllers to call provider SDKs directly. Instead, all LLM requests pass through a hardened gateway service.

Production LLM Gateway Comparison: Prototype vs. Hardened Architecture

1. Enforcing Strict Structured JSON Outputs (And Checking finish_reason)

Never parse free-form LLM text with regexes when your backend needs structured fields. Use structured output schemas (response_format: { type: "json_schema", json_schema: { strict: true, ... } }) so token sampling is constrained by a context-free grammar at generation time.

Even with strict: true, there is one edge case that still breaks production parsers: token truncation. If the model hits max_tokens before closing the JSON object, the API returns HTTP 200 OK with a half-written JSON string and finish_reason: "length". Always assert finish_reason === "stop" before parsing the payload.

2. Setting Tiered Timeouts, Jittered Backoff, and Model Fallbacks

Below is a clean, production-tested Laravel/PHP gateway pattern (the same architecture applies in TypeScript/Node or C#/.NET) that enforces strict JSON schema validation, checks finish_reason, caches deterministic calls in Redis, and falls back to a secondary model when the primary endpoint times out or returns 429/5xx:

<?php

namespace App\Services\Ai;

use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\Http;
use Illuminate\Support\Facades\Redis;
use RuntimeException;

class StructuredLlmGateway
{
    public function extractStructured(int $tenantId, string $systemPrompt, string $userInput, array $jsonSchema): array
    {
        // 1. Hard cap input length before tokenizing (~16k chars max for this workflow)
        if (mb_strlen($userInput) > 16000) {
            throw new RuntimeException('Input exceeds maximum allowed length of 16,000 characters.');
        }

        // 2. Enforce daily tenant token budget in Redis (e.g., 250,000 tokens/day)
        $budgetKey = sprintf('llm:budget:%d:%s', $tenantId, now()->utc()->format('Y-m-d'));
        $usedTokens = (int) Redis::get($budgetKey);
        if ($usedTokens >= 250000) {
            throw new RuntimeException('Daily AI processing quota reached for this workspace.');
        }

        // 3. Check deterministic Redis cache for identical (prompt + schema + input)
        $cacheKey = 'llm:v1:' . hash('sha256', json_encode([
            $systemPrompt,
            trim($userInput),
            $jsonSchema['name'] ?? 'default',
        ]));

        return Cache::remember($cacheKey, now()->addHours(24), function () use (
            $systemPrompt,
            $userInput,
            $jsonSchema,
            $budgetKey
        ) {
            $models = ['gpt-4o-mini', 'gpt-4.1-mini']; // Primary -> Fallback
            $lastError = null;

            foreach ($models as $model) {
                try {
                    return $this->callModel($model, $systemPrompt, $userInput, $jsonSchema, $budgetKey);
                } catch (\Throwable $e) {
                    $lastError = $e;
                    // Short jittered pause (150ms - 400ms) before falling back to secondary model
                    usleep(random_int(150_000, 400_000));
                }
            }

            throw new RuntimeException('All LLM models failed: ' . $lastError?->getMessage(), 0, $lastError);
        });
    }

    private function callModel(
        string $model,
        string $systemPrompt,
        string $userInput,
        array $jsonSchema,
        string $budgetKey
    ): array {
        $response = Http::withToken(config('services.openai.key'))
            ->connectTimeout(3)
            ->timeout(18)
            ->post('https://api.openai.com/v1/chat/completions', [
                'model' => $model,
                'temperature' => 0,
                'max_tokens' => 1200,
                'messages' => [
                    ['role' => 'system', 'content' => $systemPrompt],
                    ['role' => 'user', 'content' => $userInput],
                ],
                'response_format' => [
                    'type' => 'json_schema',
                    'json_schema' => [
                        'name' => $jsonSchema['name'],
                        'strict' => true,
                        'schema' => $jsonSchema['schema'],
                    ],
                ],
            ]);

        if (! $response->successful()) {
            throw new RuntimeException("LLM HTTP {$response->status()} on {$model}");
        }

        $payload = $response->json();
        $choice = $payload['choices'][0] ?? [];

        if (($choice['finish_reason'] ?? '') !== 'stop') {
            throw new RuntimeException("Incomplete LLM generation: finish_reason=" . ($choice['finish_reason'] ?? 'unknown'));
        }

        // Record token usage atomically in Redis with a 48h expiry
        $totalTokens = (int) ($payload['usage']['total_tokens'] ?? 0);
        if ($totalTokens > 0) {
            Redis::incrby($budgetKey, $totalTokens);
            Redis::expire($budgetKey, 172800);
        }

        $decoded = json_decode($choice['message']['content'] ?? '', true, 512, JSON_THROW_ON_ERROR);

        return $decoded;
    }
}

3. Offloading Slow Generations to Background Queues

Any AI workflow that summarizes a multi-page PDF, processes batch records, or chains more than one LLM call should never run inside the synchronous HTTP request-response cycle. Dispatch the job to a dedicated Redis queue (--queue=ai-high,ai-default) and let the frontend poll job status or listen over Server-Sent Events (SSE) / WebSockets.

When configuring those workers, make sure your queue retry_after timeout is always larger than your worker --timeout so a slow LLM response never causes two workers to process the same job twice—the exact configuration I detailed in running Laravel queues in production with Redis, Supervisor, retries, and failed jobs.

If you are building a Laravel-based SaaS or internal business platform that integrates AI extraction, classification, or customer support workflows, our team at Loop Origin's Laravel development practice helps teams design these pipelines cleanly from day one.

4. Estimating and Capping Token Costs Before Launch

Before deploying any new prompt to production, calculate your worst-case daily and monthly spend using real input/output token counts rather than guessing. Remember that system prompts and JSON schema definitions are billed as input tokens on every single request unless they meet the provider's prompt-prefix caching threshold. And just like strong password entropy requires measuring real bit budgets rather than rules of thumb—as I covered in how long a password should be and how password entropy works—controlling LLM costs requires measuring exact input and output tokens per tenant.

Production LLM Readiness Checklist

Run through this 6-point checklist before shipping any LLM-powered endpoint to live users:

• [ ] Enforce Strict JSON Schema + Check finish_reason: Use strict: true JSON schemas and reject any response where finish_reason !== 'stop'.
• [ ] Set Tight Connect and Read Timeouts: Configure connect_timeout (2s–3s) separately from overall read timeout (15s–20s for synchronous requests) so hung sockets fail fast.
• [ ] Move Multi-Step or Slow Calls to Redis Queues: Keep synchronous HTTP requests under a single fast model call; offload everything else to background workers.
• [ ] Cap Input Characters & Output max_tokens: Validate maximum input character length at the controller boundary before calling the API.
• [ ] Cache Deterministic Extraction & Classification: Hash normalized inputs with your prompt version in Redis (temperature = 0) so identical payloads cost $0 on repeat hits.
• [ ] Enforce Per-Tenant Daily Token Budgets: Track usage.total_tokens in Redis with INCRBY and alert your team on Slack/email at 70% and 100% thresholds.

Frequently Asked Questions (FAQ)

Why did my LLM return invalid JSON even though I enabled JSON mode?

Basic JSON mode (response_format: { type: "json_object" }) only guarantees that the output is syntactically valid JSON if generation finishes normally—it does not enforce your specific field names or types, and it still returns truncated, broken JSON if the response hits max_tokens (finish_reason: "length"). Use type: "json_schema" with strict: true and always verify finish_reason === "stop".

Should I retry 429 Too Many Requests errors inside a synchronous web request?

At most once, after a short jittered delay (150ms–400ms), or immediately route the retry to a secondary model family or deployment endpoint that has its own separate rate-limit bucket. Sleeping for 10+ seconds inside a synchronous web worker will quickly exhaust your web server's connection pool.

How do I prevent prompt injection when passing user-submitted text to an LLM?

First, keep instructions strictly in the system (or developer) message and wrap untrusted user data inside clear delimiter tags in the user message. Second, never allow model output to execute raw SQL, shell commands, or unvalidated tool calls; restrict output to a strict JSON schema with enumerated values and validate every field on your backend before acting on it.

Does setting temperature: 0 guarantee 100% identical outputs across every API call?

It makes outputs nearly deterministic, though minor floating-point variations across GPU batches in mixture-of-experts (MoE) architectures can occasionally change a token. That is why caching the validated JSON output in Redis by input hash is both faster and more consistent than re-calling the API.

When should I switch from synchronous API calls to streaming (SSE)?

Whenever the user is waiting interactively for prose output longer than 150–200 tokens (such as a chat assistant or draft generator). Streaming reduces time-to-first-token (TTFT) to under 600 milliseconds so the UI feels instant even if the full generation takes 6 seconds. For structured JSON extraction where the backend needs the complete object before continuing, use a fast small model or a background job instead.