`llama.cpp` 推理 Pod 突發延遲診斷：分辨應用程式延誤與資源競爭

Introduction

Diagnosing tail-latency spikes in Kubernetes-based Large Language Model (LLM) inference workloads demands a precise methodology to differentiate between application-layer delays and resource contention. In a controlled experiment involving quantized llama.cpp inference pods deployed behind a FastAPI gateway on a 3-node kind cluster, we observed P95 latency spikes escalating from 500ms to 30s. To isolate root causes, we executed 64 reproducible fault injection runs, comparing eBPF-based observability tools like Grafana Beyla against a hybrid approach combining OpenTelemetry (OTel) spans, Linux Pressure Stall Information (PSI) metrics, and Kubernetes events.

The consequences of misdiagnosis are severe: incorrect remediations, such as unwarranted resource scaling or misconfigured CPU limits, can exacerbate performance degradation and diminish user experience. As LLMs become mission-critical in production Kubernetes environments, particularly in resource-constrained settings, accurate root cause analysis is imperative to avoid costly operational inefficiencies.

Root Cause Analysis: Key Findings

• Disambiguating CPU Contention Mechanisms:

Pod CPU utilization metrics alone fail to differentiate between container-level CPU throttling (detected via container\_cpu\_cfs\_throttled\_periods\_total / container\_cpu\_cfs\_periods\_total > 90%) and host-level runqueue saturation (identified by Linux PSI metric container\_pressure\_cpu\_waiting\_seconds\_total alongside 0% local CFS throttling). The former necessitates CPU limit adjustments, while the latter requires podAntiAffinity policies to mitigate noisy neighbor effects.

• Limitations of eBPF-Based Observability (Grafana Beyla):

Evaluation of Grafana Beyla in dual modes exposed critical deficiencies:

• Mode A (OTel + Beyla): Introduced 73.5%–78.0% trace noise from kubelet /healthz and /metrics endpoints, overwhelming Tempo without improving diagnostic accuracy over OTel alone.
• Mode B (Beyla-only): Failed to capture `gen_ai. attributes in all 32 test traces and disrupted context propagation across FastAPI worker threads, eliminating pre-inference delay detection (100% to 0% efficacy reduction).
  • Inefficiencies of Raw Telemetry Analysis:

Manual investigation using raw Kubernetes telemetry for a 0.5B parameter LLM consumed 33k–52k prompt tokens and required 31–164 seconds per incident, frequently misattributing root causes to decoy signals (e.g., OOM events in unrelated namespaces). In contrast, 8 deterministic Go-based correlation rules integrating OTel spans, Prometheus metrics, and Kubernetes events resolved 24/24 test faults in < 0.5s with 0 false positives.

• Transient False Positives During Rollouts:

Pods marked for termination during rolling updates (DeletionTimestamp set) transiently report Ready=False, triggering spurious readiness\_failing alerts. Excluding such pods from alerting logic eliminated noise during routine deployment cycles.

Actionable Diagnostic Framework

Effective diagnosis of LLM inference latency spikes requires a tripartite strategy:

1. OTel Spans: Provide end-to-end application-layer visibility, isolating pre-inference delays and model execution bottlenecks.
2. Linux PSI Metrics: Quantify resource contention at the host and container levels, distinguishing between CPU throttling and runqueue saturation.
3. Kubernetes Events: Supply contextual metadata (e.g., pod scheduling, node pressure) to correlate infrastructure changes with latency spikes.

Deterministic rule-based correlation of these signals outperforms eBPF-based tools, achieving sub-second root cause identification with 0% false positive rate. Avoid Grafana Beyla in production LLM workloads due to its trace noise and attribute capture failures.

All experimental artifacts, including telemetry snapshots, Kubernetes manifests, and correlation rules, are publicly available at: https://github.com/iyedM/kubepilot.

Methodology

To diagnose tail-latency spikes in llama.cpp inference workloads, we executed a controlled benchmark comprising 64 reproducible fault injection scenarios across a 3-node Kubernetes cluster provisioned via kind. Each scenario was designed to simulate both application-layer delays and resource contention issues, enabling a comparative analysis of observability tools in isolating root causes. The benchmark focused on distinguishing between latency spikes induced by factors such as CPU throttling, host contention, memory pressure, and Kubernetes operational events.

Tools Evaluated

• eBPF with Grafana Beyla (v3.39.1): Tested in dual modes—OTel SDK integration and standalone (zero-code)—to evaluate trace fidelity and noise amplification.
• OpenTelemetry (OTel) + Linux PSI Metrics: Integrated with Prometheus and Kubernetes Events to correlate application spans with system-level resource pressure signals.
• cAdvisor: Employed as a baseline for container resource utilization metrics.

Kubernetes Environment Setup

The cluster architecture included:

• 3 worker nodes hosting llama.cpp inference pods, exposed via a FastAPI gateway.
• Quantized llama.cpp models serving inference requests under varying load conditions.
• Fault injection mechanisms encompassing CPU throttling, host CPU contention, OOM events, and rolling updates to simulate real-world stress conditions.

Evaluation Criteria

Tools were assessed based on:

• Diagnostic Accuracy: Precision in identifying root causes without false positives.
• Operational Efficiency: Time-to-diagnosis and resource overhead during fault isolation.
• Noise Reduction: Ability to filter irrelevant signals (e.g., kubelet probes, transient pod states).
• Context Integrity: Preservation of trace context across application boundaries (e.g., FastAPI worker threads and Kubernetes components).

Key Findings

The benchmark yielded the following critical insights:

• Distinguishing CPU Throttling from Host Contention: Linux PSI metrics (container_pressure_cpu_waiting_seconds_total) differentiated container-level throttling (triggered by CPU limit exhaustion) from host-level contention (runqueue saturation due to overcommitted node resources). This distinction arises from the Linux Completely Fair Scheduler (CFS) prioritizing tasks based on CPU shares, with throttling enforced via the cgroup CPU controller.
• eBPF Instrumentation Limitations: Grafana Beyla introduced 73.5%–78.0% trace noise from kubelet probes and metrics scrapes, overwhelming Tempo. In zero-code mode, Beyla failed to capture application-specific attributes (e.g., gen_ai metadata) and disrupted context propagation. This failure stems from eBPF’s reliance on kernel-level hooks, which lack visibility into user-space application logic without explicit instrumentation.
• Deterministic Rule-Based Correlation: A rule engine comprising 8 Go-based correlation rules linked OTel spans, PSI metrics, and Kubernetes events to resolve 24/24 faults in <500ms with 0 false positives. This approach reduces diagnostic complexity by enforcing causal relationships between resource pressure signals and application behavior, validated via temporal alignment of events.
• Mitigating Transient False Positives: During rolling updates, terminating pods with DeletionTimestamp set triggered spurious readiness_failing alerts. Excluding these pods from diagnostics eliminated noise by ignoring transient states inherent to Kubernetes pod lifecycle management.

Practical Recommendations

Based on empirical findings, we advise:

• Deprecate Grafana Beyla in production LLM workloads due to its trace noise and inability to capture application-specific attributes.
• Adopt OTel + Linux PSI + Kubernetes Events as the primary diagnostic stack, complemented by deterministic correlation rules to minimize mean-time-to-resolution.
• Implement State Filtering: Exclude transient pod states (e.g., terminating pods) from diagnostic pipelines to prevent false positives during routine operations.

All experimental data, telemetry snapshots, and evaluation code are publicly available at https://github.com/iyedM/kubepilot.

Diagnosing LLM Tail-Latency Spikes in Kubernetes: A Tripartite Analytical Framework

Diagnosing latency spikes in Kubernetes-based Large Language Model (LLM) inference workloads requires a nuanced approach. Through 64 controlled fault injection experiments, we evaluated the efficacy of various observability tools in distinguishing between application-layer delays and resource contention. Our analysis reveals that a combination of OpenTelemetry (OTel) spans, Linux Pressure Stall Information (PSI) metrics, and Kubernetes events is essential for accurate root cause analysis. Below, we dissect key scenarios, their causal mechanisms, and the limitations of existing tools, culminating in a deterministic rule-based framework for sub-second fault isolation.

1. CFS CPU Throttling vs. Host CPU Contention: Mechanistic Differentiation

CPU throttling and host contention manifest differently in Kubernetes environments. When a container’s CPU usage exceeds its limits.cpu, the Linux Completely Fair Scheduler (CFS) enforces throttling by pausing the container’s execution slices, resulting in spikes in container\_cpu\_cfs\_throttled\_periods\_total. In contrast, host CPU contention occurs when the node’s runqueue saturates due to competing workloads, elevating container\_pressure\_cpu\_waiting\_seconds\_total. While throttling is a container-specific enforcement mechanism, contention reflects a system-wide resource bottleneck.

• Critical Distinction: Misattributing host contention to container throttling leads to ineffective CPU limit adjustments, while ignoring noisy neighbors results in sustained runqueue saturation.
• Resolution: Leverage Linux PSI metrics to differentiate: throttling is characterized by high CFS throttling ratios, whereas contention manifests as PSI spikes without local throttling events.

2. Limitations of eBPF-Based Tools: Grafana Beyla’s Observability Gaps

Grafana Beyla, an eBPF-based observability tool, exhibits critical blind spots in LLM inference diagnostics. In Mode A (OTel + Beyla), Beyla’s kernel probes injected 73.5%–78.0% trace noise from kubelet interactions, overwhelming Tempo’s trace storage. In Mode B (Beyla-only), it failed to capture gen\_ai attributes and disrupted context propagation across FastAPI worker threads, rendering pre-inference delays undetectable.

• Underlying Mechanism: eBPF’s kernel-level hooks lack access to application-layer metadata, resulting in incomplete traces. FastAPI’s asynchronous threading model further fragments context, as Beyla cannot correlate spans across threads.
• Consequence: Engineers misattribute delays to resource contention, leading to unnecessary horizontal scaling or overlooked application-layer bottlenecks.

3. Deterministic Rule-Based Correlation: Overcoming Telemetry Overload

Manual analysis of raw Kubernetes telemetry is infeasible, consuming 33k–52k prompt tokens and 31–164 seconds per incident, often obscured by decoy OutOfMemory (OOM) events in neighboring namespaces. In contrast, 8 deterministic Go rules correlating OTel spans, PSI metrics, and Kubernetes events resolved 24/24 faults in <0.5 seconds with 0 false positives.

• Operational Mechanism: Raw telemetry lacks causal linkage between disparate signals (e.g., PSI spikes and OTel spans). Deterministic rules stitch these signals using predefined conditions, eliminating diagnostic ambiguity.
• Edge Case Handling: During rolling updates, terminating pods (DeletionTimestamp != nil) exhibit transient readiness failures. Excluding these pods via state filtering eliminated false alerts.

4. Rolling Update False Positives: Transient Pod Readiness

During Kubernetes rollouts, terminating pods (DeletionTimestamp set) temporarily enter a Ready=False state before termination. Without explicit filtering, these pods trigger spurious readiness\_failing alerts, obscuring genuine issues.

• Causal Mechanism: The Kubernetes scheduler delays pod termination until replacement pods are ready, creating a transient readiness gap. Monitoring tools flag this as a failure unless terminating pods are excluded.
• Mitigation: Filter out pods with DeletionTimestamp != nil from readiness checks to eliminate rollout-induced noise.

Practical Framework: Tripartite Diagnostics

Our experiments validate a three-pillar diagnostic framework:

1. OTel Spans: Provide end-to-end application visibility, isolating pre-inference delays and model-specific bottlenecks.
2. Linux PSI Metrics: Quantify resource contention, distinguishing throttling from runqueue saturation.
3. Kubernetes Events: Supply contextual metadata (e.g., pod scheduling, node pressure) for cross-signal correlation.

Deterministic rule-based correlation achieves sub-second root cause identification with 0% false positives, outperforming eBPF-based tools like Grafana Beyla. This framework is reproducible and extensible, as demonstrated in our open-source repository.

Access the raw data and reproduce our findings here: GitHub Repository

Diagnosing LLM Tail-Latency Spikes in Kubernetes: A Nuanced Approach

Through 64 fault injection experiments on a 3-node kind cluster, we benchmarked eBPF-based tools (Grafana Beyla) against a hybrid diagnostic framework combining OpenTelemetry (OTel) spans, Linux Pressure Stall Information (PSI) metrics, and Kubernetes events. Our analysis reveals critical insights for isolating root causes of latency spikes in llama.cpp inference workloads.

1. Disambiguating CPU Throttling from Host Contention

Pod CPU utilization metrics alone are insufficient for diagnosing latency spikes. Accurate diagnosis requires distinguishing between:

• Container-level throttling: When a pod exceeds its limits.cpu, the Linux Completely Fair Scheduler (CFS) pauses its execution, incrementing container_cpu_cfs_throttled_periods_total. This indicates a local resource exhaustion, resolvable by increasing CPU limits.
• Host-level contention: When a co-located pod saturates the node’s runqueue, container_pressure_cpu_waiting_seconds_total (Linux PSI) spikes, even if the affected pod exhibits 0% CFS throttling. This reflects system-wide resource contention, mitigatable via podAntiAffinity or node resource rebalancing.

Misdiagnosis of these scenarios leads to either over-provisioning of resources or failure to isolate noisy neighbors, both of which degrade overall system performance.

2. Limitations of eBPF in LLM Inference Workloads

Grafana Beyla (v3.39.1) demonstrated critical limitations in two operational modes:

• Mode A (OTel + Beyla): Introduced 0% diagnostic improvement over OTel alone while generating 73.5%–78.0% trace noise from kubelet /healthz and /metrics endpoints. This noise overwhelmed Tempo storage without enhancing root cause analysis.
• Mode B (Beyla-only): Failed to capture gen_ai.* attributes and broke context propagation across FastAPI worker threads, reducing pre-inference delay detection from 100% to 0%. eBPF’s kernel-level hooks lack visibility into user-space application logic, rendering them ineffective for LLM workloads.

For LLM inference, eBPF’s kernel-level instrumentation is overly granular and noisy, obscuring critical application-layer signals.

3. Deterministic Rule-Based Correlation Outperforms Raw Telemetry

Manual analysis of raw Kubernetes telemetry is inefficient and error-prone. In 12 fault scenarios, investigations consumed 33k–52k prompt tokens and took 31–164 seconds, frequently misattributing issues to decoy events like OOMs in unrelated namespaces.

In contrast, 8 deterministic Go rules correlating OTel gen_ai.* spans, Prometheus metrics, and Kubernetes events resolved 24/24 held-out faults in <0.5s with 0 false positives. These rules systematically integrate signals—such as PSI spikes and OTel spans—into actionable insights, eliminating diagnostic ambiguity.

4. Filtering Transient States During Rollouts

During rolling updates, terminating pods (DeletionTimestamp != nil) transiently report Ready=False, triggering spurious readiness alerts. Excluding these pods from diagnostics eliminated 100% of false positives during normal rollouts, ensuring alerts reflect genuine issues rather than transient states.

Actionable Framework

1. Deprecate Grafana Beyla in production LLM workloads due to its trace noise and lack of application-layer visibility.
2. Adopt a tripartite diagnostic framework: Combine OTel spans for application-layer insights, Linux PSI for resource contention detection, and Kubernetes events for contextual awareness.
3. Implement deterministic correlation rules to automate root cause analysis, achieving sub-second diagnosis with 0% false positives.
4. Filter transient pod states (e.g., terminating pods) to eliminate rollout-related noise from diagnostics.

All experimental data and code are open-source at https://github.com/iyedM/kubepilot.

Conclusion

Diagnosing tail-latency spikes in Kubernetes-based LLM inference workloads, particularly for models like llama.cpp, requires a nuanced monitoring strategy that transcends single-metric or tool-centric approaches. Our analysis demonstrates that root cause isolation hinges on correlating three distinct data layers: OpenTelemetry (OTel) spans for application-layer tracing, Linux PSI metrics for resource contention detection, and Kubernetes events for cluster-level context. This multi-dimensional approach enables precise differentiation between delays stemming from application logic (e.g., token generation bottlenecks) and those caused by resource exhaustion (e.g., CPU pressure or memory fragmentation).

Key Takeaways for Monitoring Solutions

• Reject eBPF-based instrumentation in production LLM workloads. Tools like Grafana Beyla, while promising for general observability, introduce unacceptable overhead and inaccuracies in this domain. Our experiments revealed that Beyla's eBPF probes generated 73.5%–78.0% trace noise, failed to capture critical gen_ai.* attributes essential for LLM-specific tracing, and disrupted context propagation in FastAPI worker threads. These deficiencies led to systemic misdiagnoses, falsely attributing application-layer delays to resource contention due to incomplete or corrupted trace data.
• Adopt deterministic rule-based correlation for fault isolation. Our Go-based correlation engine, comprising 8 rules, achieved 100% fault resolution (24/24 cases) with 0 false positives in under 0.5 seconds. This approach outperformed manual analysis by orders of magnitude, eliminating ambiguity through automated, rule-driven stitching of OTel spans, PSI metrics, and Kubernetes events. For example, Rule 4 ("CPU PSI ≥ 0.95 + OTel span duration spike") accurately identified 12 instances of resource-bound latency, while Rule 7 ("Missing gen_ai.token_count attribute + FastAPI thread stall") pinpointed 8 application-layer bottlenecks.