Explaining AI Agents Through Execution Traces

• URL: http://arxiv.org/abs/2609.06063v1
• Date: Sat, 05 Sep 2026 12:45:56 GMT
• Title: Explaining AI Agents Through Execution Traces
• Abstract summary: We present a post-hoc XAI framework that transforms a lengthy agent's execution trace into a structured report and a faithful natural-language explanation.<n>Because it relies solely on execution traces, the framework applies across different agent architectures, environments, and tasks.
• Score: 1.8862116040247214
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI Agents are increasingly deployed in real-world settings, where they interact with external tools and make sequential decisions with limited human oversight. This creates a pressing need for reliable and auditable explanations of what an agent did and why. However, traditional Explainable AI (XAI) methods fall short of providing the process-level transparency required for such interactive, multi-step systems, motivating a paradigm shift toward approaches specifically designed for AI Agents. To address this gap, we present a post-hoc XAI framework that transforms a lengthy agent's execution trace into a structured report and a faithful natural-language explanation explicitly grounded in its observable behavior. Because it relies solely on execution traces, the framework applies across different agent architectures, environments, and tasks. Human and automated evaluations across multiple benchmarks and architectures show that our framework produces high-quality, trace-faithful explanations while reliably identifying unsupported claims, unjustified actions, and evidence gaps, outperforming naive LLM-generated explanations.

Related papers

• Declarative Skills for AI Agents in Knowledge-Grounded Tool-Use Workflows [1.9573380763700712]
  We study orchestration mechanisms for tool-using AI agents in realistic customer-service errors over an unstructured knowledge base.<n>We compare (i) a DeclarativeAgent that reads three domain-specific skill files at inference time and decides its own control flow, (ii) an ImperativeAgent based on a programmatic state machine with explicit phases, and (iii) an unscaffolded baseline agent modeled after the $$-Knowledge benchmark agent.
  arXiv  Detail & Related papers  (2026-06-05T05:38:51Z)
• RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMs [55.44696796126619]
  Earth Observation systems are designed to support domain experts who often express their requirements through vague natural language.<n>A practical EO AI system must bridge the gap between ambiguous human queries and the appropriate multi-granularity visual analysis tasks.<n>We propose RemoteAgent, an agentic framework that strategically respects the capability intrinsic boundaries of MLLMs.
  arXiv  Detail & Related papers  (2026-04-09T03:40:46Z)
• XAI for Coding Agent Failures: Transforming Raw Execution Traces into Actionable Insights [0.0]
  We present a systematic explainable AI (XAI) approach that transforms raw agent execution traces into structured, human-interpretable explanations.<n>We demonstrate that our approach enables users to identify failure root causes 2.8 times faster and propose correct fixes with 73% higher accuracy compared to raw execution traces.
  arXiv  Detail & Related papers  (2026-03-06T06:18:20Z)
• Beyond the 'Diff': Addressing Agentic Entropy in Agentic Software Development [42.371764229953165]
  We propose a process-oriented explainability framework that exposes how agentic decisions unfold across time, tool calls, and architectural boundaries.<n>Our approach provides intent-level telemetry that complements, rather than replaces, existing review practices.<n>By treating cognitive drift as a first-class concern alongside code quality, our framework supports the minimum level of human comprehension required for agentic oversight to remain substantive.
  arXiv  Detail & Related papers  (2026-03-03T12:55:28Z)
• The Why Behind the Action: Unveiling Internal Drivers via Agentic Attribution [63.61358761489141]
  Large Language Model (LLM)-based agents are widely used in real-world applications such as customer service, web navigation, and software engineering.<n>We propose a novel framework for textbfgeneral agentic attribution, designed to identify the internal factors driving agent actions regardless of the task outcome.<n>We validate our framework across a diverse suite of agentic scenarios, including standard tool use and subtle reliability risks like memory-induced bias.
  arXiv  Detail & Related papers  (2026-01-21T15:22:21Z)
• Towards Responsible and Explainable AI Agents with Consensus-Driven Reasoning [4.226647687395254]
  This paper presents a Responsible(RAI) and Explainable(XAI) AI Agent Architecture for production-grade agentic based on multi-model consensus and reasoning-layer governance.<n>In the proposed design, a consortium of heterogeneous LLM and VLM agents independently generates candidate outputs from a shared input context.<n>A dedicated reasoning agent then performs structured consolidation across these outputs, enforcing safety and policy constraints, mitigating hallucinations and bias, and producing auditable, evidence-backed decisions.
  arXiv  Detail & Related papers  (2025-12-25T14:49:25Z)
• AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answering [51.07491603393163]
  tAgent is a framework that formulates multi-agent QA as a knowledge-graph-guided routing problem supervised by empirical performance signals.<n>By leveraging soft supervision and weighted aggregation of agent outputs, Agent learns principled collaboration schemes that capture the complementary strengths of diverse agents.
  arXiv  Detail & Related papers  (2025-10-06T23:20:49Z)
• Agent-as-a-Judge: Evaluate Agents with Agents [61.33974108405561]
  We introduce the Agent-as-a-Judge framework, wherein agentic systems are used to evaluate agentic systems. This is an organic extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving process. We present DevAI, a new benchmark of 55 realistic automated AI development tasks.
  arXiv  Detail & Related papers  (2024-10-14T17:57:02Z)
• Gödel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement [112.04307762405669]
  G"odel Agent is a self-evolving framework inspired by the G"odel machine.<n>G"odel Agent can achieve continuous self-improvement, surpassing manually crafted agents in performance, efficiency, and generalizability.
  arXiv  Detail & Related papers  (2024-10-06T10:49:40Z)
• Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoning [50.47568731994238]
  Key method for creating Artificial Intelligence (AI) agents is Reinforcement Learning (RL) This paper presents a general framework model for integrating and learning structured reasoning into AI agents' policies.
  arXiv  Detail & Related papers  (2023-12-22T17:57:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.