Ekka: Automated Diagnosis of Silent Errors in LLM Inference

• URL: http://arxiv.org/abs/2606.04594v1
• Date: Wed, 03 Jun 2026 08:32:13 GMT
• Title: Ekka: Automated Diagnosis of Silent Errors in LLM Inference
• Abstract summary: Diagnosing silent errors is notoriously difficult due to the substantial semantic gap between the high-level symptoms and the low-level root causes.<n>We propose Ekka, an automated diagnosis system that identifies root causes by systematically aligning and comparing intermediate execution states.<n>Ekka shows 80% pass@1 diagnosis accuracy and 88% pass@5 diagnosis accuracy, outperforming state-of-the-art systems.
• Score: 13.232208959993331
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM serving frameworks are quickly evolving with a complex software stack and a vast number of optimizations. The rapid development process can introduce silent errors where output quality silently degrades without any explicit error signals. Diagnosing silent errors is notoriously difficult due to the substantial semantic gap between the high-level symptoms and the low-level root causes. We observe that diagnosis of silent errors can be effectively framed as a differential debugging problem by leveraging the existence of semantically correct reference implementations. We propose Ekka, an automated diagnosis system that identifies root causes by systematically aligning and comparing intermediate execution states between a target and a reference framework. We constructed a benchmark of real-world silent errors from popular serving frameworks, where Ekka shows 80% pass@1 diagnosis accuracy and 88% pass@5 diagnosis accuracy, outperforming state-of-the-art systems. Ekka also diagnoses 4 new silent errors from serving frameworks, all of which have been confirmed by the developers.

Related papers

• DCFA: Dual-view Causal-inspired Attribution for Failure Reasoning in LLM-based Multi-agent Systems [61.11435469308521]
  We propose DCFA, a training-free framework for failure attribution in large language model systems.<n> DCFA integrates a global module that constructs structured causal-inspired dependency graphs from system traces.<n>Experiments show that DCFA improves step-level accuracy by up to 8.27% over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-09-04T05:29:31Z)
• Hierarchical Fault Localization for Autonomous Driving Systems with Hypothesis Validation and Intent Analysis [14.387618464803774]
  We present HINT, a two-phase framework for hierarchical ADS fault localization based on hypothesis validation and intent analysis.<n>In Phase I, HINT transforms failure-triggering execution recordings into multi-modal abstractions.<n>In Phase II, it reconstructs design-side intent and implementation-side behavior, then localizes suspicious code through reliability-aware consistency checking.
  arXiv  Detail & Related papers  (2026-07-14T10:17:01Z)
• Can Large Language Models Generate Observability-Aware Code? [51.30941152293301]
  We present a systematic study of observability in agent-generated systems.<n>Our results reveal a consistent gap between diagnostic semantics at the source level and fault signals at runtime.<n>We introduce an observability-oriented skill, which can serve as a guidance to improve both diagnostic semantics and fault-signal exposure.
  arXiv  Detail & Related papers  (2026-07-07T03:20:52Z)
• REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces [10.98846592145896]
  Large language model (LLM) agents solve complex tasks through long plan-and-execution traces, yet the ability to locate errors in a completed traces still lags far behind.<n>We propose methodname, a method that closes this gap by diagnosing a candidate error step, testing it through controlled replay with a diagnosis-specific patch, and using the verified outcome flip as contrastive evidence to refine the final attribution.
  arXiv  Detail & Related papers  (2026-06-08T06:11:57Z)
• Proactive for Uncertainty: Cause-Aware Error Diagnosis and Interactive Clarification for Spoken Dialogue Systems [40.265250435329456]
  We propose a cause-aware error recovery paradigm that fundamentally rethinks robustness in SDS.<n>Unlike traditional confidence filtering, we introduce a suite of small precision-focused detectors.<n>This fine-grained diagnostic intelligence empowers the LLM to orchestrate targeted, multi-turn clarification strategies.
  arXiv  Detail & Related papers  (2026-05-25T03:57:38Z)
• Towards Self-Improving Error Diagnosis in Multi-Agent Systems [31.04613892300063]
  We introduce ErrorProbe, a self-improving framework for semantic failure attribution.<n>ErrorProbe identifies responsible agents and the originating error step.<n>It maintains a verified episodic memory that updates only when error patterns are confirmed by executable evidence.
  arXiv  Detail & Related papers  (2026-04-19T23:13:05Z)
• Kunlun Anomaly Troubleshooter: Enabling Kernel-Level Anomaly Detection and Causal Reasoning for Large Model Distributed Inference [15.448826510384302]
  Anomaly troubleshooting for large model distributed inference (LMDI) remains a critical challenge.<n>We introduce Kunlun Anomaly Troubleshooter (KAT), the first anomaly troubleshooting framework tailored for LMDI.
  arXiv  Detail & Related papers  (2025-11-08T11:53:08Z)
• InspectCoder: Dynamic Analysis-Enabled Self Repair through interactive LLM-Debugger Collaboration [71.18377595277018]
  Large Language Models (LLMs) frequently generate buggy code with complex logic errors that are challenging to diagnose.<n>We present InspectCoder, the first agentic program repair system that empowers LLMs to actively conduct dynamic analysis via interactive debugger control.
  arXiv  Detail & Related papers  (2025-10-21T06:26:29Z)
• Where LLM Agents Fail and How They can Learn From Failures [62.196870049524364]
  Large Language Model (LLM) agents have shown promise in solving complex, multi-step tasks.<n>They amplify vulnerability to cascading failures, where a single root-cause error propagates through subsequent decisions.<n>Current systems lack a framework that can comprehensively understand agent error in a modular and systemic way.<n>We introduce the AgentErrorTaxonomy, a modular classification of failure modes spanning memory, reflection, planning, action, and system-level operations.
  arXiv  Detail & Related papers  (2025-09-29T18:20:27Z)
• Towards Automated Error Discovery: A Study in Conversational AI [48.735443116662026]
  We introduce Automated Error Discovery, a framework for detecting and defining errors in conversational AI.<n>We also propose SEEED (Soft Clustering Extended-Based Error Detection), as an encoder-based approach to its implementation.
  arXiv  Detail & Related papers  (2025-09-13T14:53:22Z)
• GALA: Can Graph-Augmented Large Language Model Agentic Workflows Elevate Root Cause Analysis? [9.394057684388027]
  This paper introduces GALA, a novel framework for root cause analysis in microservice systems.<n> evaluated on an open-source benchmark, GALA achieves substantial improvements over state-of-the-art methods.<n>We show that GALA bridges the gap between automated failure diagnosis and practical incident resolution.
  arXiv  Detail & Related papers  (2025-08-17T19:12:05Z)
• Hide and Seek with LLMs: An Adversarial Game for Sneaky Error Generation and Self-Improving Diagnosis [51.88592148135258]
  We propose Hide and Seek Game (HSG), a dynamic adversarial framework for error generation and diagnosis.<n>HSG involves two adversarial roles: Sneaky, which "hides" by generating subtle, deceptive reasoning errors, and Diagnosis, which "seeks" to accurately detect them.<n> Experiments on several math reasoning tasks show that HSG significantly boosts error diagnosis, achieving 16.8%--31.4% higher accuracy than baselines like GPT-4o.
  arXiv  Detail & Related papers  (2025-08-05T12:45:21Z)
• Causal Disentanglement Hidden Markov Model for Fault Diagnosis [55.90917958154425]
  We propose a Causal Disentanglement Hidden Markov model (CDHM) to learn the causality in the bearing fault mechanism. Specifically, we make full use of the time-series data and progressively disentangle the vibration signal into fault-relevant and fault-irrelevant factors. To expand the scope of the application, we adopt unsupervised domain adaptation to transfer the learned disentangled representations to other working environments.
  arXiv  Detail & Related papers  (2023-08-06T05:58:45Z)
• Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors [105.12462629663757]
  In this work, we aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. We compare performance of state-of-the-art factuality metrics, including recent ChatGPT-based metrics, on this stratified benchmark and show that their performance varies significantly across different types of summarization models.
  arXiv  Detail & Related papers  (2022-05-25T15:26:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.