From Agent Traces to Trust: Evidence Tracing and Execution Provenance in LLM Agents

• URL: http://arxiv.org/abs/2606.04990v1
• Date: Wed, 03 Jun 2026 15:12:47 GMT
• Title: From Agent Traces to Trust: Evidence Tracing and Execution Provenance in LLM Agents
• Abstract summary: Large language model (LLM)-based agents increasingly solve complex tasks by interacting with external tools, retrieval systems, memory modules, environments, and other agents.<n>Final-answer accuracy alone cannot explain how an output was produced, which evidence supported each claim, whether tool calls were justified, how memory influenced later decisions, or where execution failures originated.<n>Evidence tracing and execution provenance address this gap by modeling how retrieved evidence, tool outputs, memory items, environment observations, intermediate claims, actions, and final answers are connected throughout agent execution.
• Score: 20.651400118242346
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model (LLM)-based agents increasingly solve complex tasks by interacting with external tools, retrieval systems, memory modules, environments, and other agents. These capabilities expand agent autonomy, but also make agent behavior harder to verify, debug, and audit. Final-answer accuracy alone cannot explain how an output was produced, which evidence supported each claim, whether tool calls were justified, how memory influenced later decisions, or where execution failures originated. Evidence tracing and execution provenance address this gap by modeling how retrieved evidence, tool outputs, memory items, environment observations, intermediate claims, actions, and final answers are connected throughout agent execution. This survey provides a systematic review and conceptual framework for evidence tracing and execution provenance in LLM agents. We organize related work around a unified provenance perspective that connects retrieval grounding, claim support, tool-use safety, memory lineage, observability, debugging, audit, and recovery. We introduce a taxonomy covering trace sources, evidence and execution units, provenance relations, tracing granularity and timing, representation forms, and trust functions. We review key methodological directions, including provenance representation, evidence attribution, tool-use provenance, runtime guardrails, provenance-bearing memory, trace-based observability, and failure diagnosis. We also map existing benchmarks, datasets, and evaluation metrics to provenance-related capabilities, and discuss how evaluation can move from final-answer correctness toward process-level accountability. Finally, we outline open challenges, including unified trace schemas, claim-level and semantic provenance, provenance-aware safety mechanisms, realistic execution-trace benchmarks, recovery-oriented evaluation, and privacy-aware audit infrastructure.

Related papers

• Terminal Agents: A Survey of AI Agents in Command-Line Environments [102.48222742145474]
  Large language model agents increasingly act through terminals, yet existing surveys disperse terminal-mediated behavior across software engineering, tool use, and computer-use research.<n>We regard terminal agents as systems whose dominant progress-bearing action--observation loop is mediated by terminal command execution, textual feedback, and stateful environment interaction.<n>Our synthesis shows that realized behavior is jointly shaped by the model, interface, harness, runtime, and environment.
  arXiv  Detail & Related papers  (2026-08-20T18:16:44Z)
• LEDGER: Claim-to-Evidence Trace Graphs for Auditing LLM Agents [2.9951905451976106]
  LEDGER builds trace graphs over observed agent sessions.<n>We show how the resulting traces expose workflow decisions, artifact lineage, repair steps, validation steps, and claim-support paths for evidence-centered audit.
  arXiv  Detail & Related papers  (2026-08-19T00:10:04Z)
• From Sequence to Structure: Relational Uncertainty Propagation for LLM Agents [75.69180947909148]
  We propose a trajectory-level UQ framework for large language model (LLM) agents.<n>RuPA represents an execution history as a directed trajectory graph in which reasoning states, tool interactions, and environment feedback are nodes connected by temporal and semantic dependency edges.<n>We evaluate RUPA on representative agent benchmarks, including $$-2, Terminal-Bench-2, and GAIA, using 6 open-source LLMs spanning multiple model families.
  arXiv  Detail & Related papers  (2026-08-17T01:40:14Z)
• Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model [0.0]
  Book synthesizes 164 scholarly works, 100 practitioner records, 29 benchmark records, and 17 author-system case records.<n>Written as a structured multivocal review, targeted update audits, software-engineering coverage analysis, and distributed-systems evidence synthesis.<n>Book contributes a versioned catalog of 206 reliability records, including 56 developed in depth, plus 13 research leads.
  arXiv  Detail & Related papers  (2026-08-14T01:34:25Z)
• Formal Verification of Agentic Systems over Operational Data [59.98246281888422]
  Agentic systems driven by large language models (LLMs) are increasingly deployed in real-world where they act on persistent operational data.<n>Existing approaches do not provide such system-level guarantees.<n>We study here the verification of agentic systems comprising a single LLM and a tool orchestration harness over relational operational data.
  arXiv  Detail & Related papers  (2026-08-04T13:01:30Z)
• PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution [49.776611937968]
  We present PhyAgentOS, a foundation delivering scheduling, verification, memory, benchmarking, and safety as system-level services.<n>Its Session-Centered treats a session, not an action, as the minimum unit of scheduling, compatibility, supervised execution, evidence collection, and acceptance.<n>SessionVerifier distinguishes execution termination from semantic task completion via evidence-grounded verdicts of success, failure, or replan.<n> Benchmarking reuses the deployment session and verification path, so results trace to real execution.
  arXiv  Detail & Related papers  (2026-07-18T04:46:53Z)
• QwenPaw-Data: Bridging Facts, Methodology, and Execution for Autonomous Enterprise Data Analytics [45.48758200829718]
  We introduce QwenPaw-Data, an agentic data system designed for enterprise intelligent data analysis.<n>QwenPaw-Data consolidates heterogeneous assets into reusable, governable, and evolvable analysis assets, then turns natural-language requests into end-to-end analytical spanning data understanding, retrieval, analysis, report generation, and decision support.
  arXiv  Detail & Related papers  (2026-07-13T02:38:00Z)
• From Question Answering to Task Completion: A Survey on Agent System and Harness Design [67.8241530947325]
  This survey examines LLM-based agents through a model-harness lens.<n>We first clarify the functional definition of agents and the implementation view of an LLM-based agent as a foundation model and an execution harness.<n>We then analyze the limits of model-centric scaling, trace four paradigms of agent engineering, and decompose the execution harness into six coupled runtime responsibilities.
  arXiv  Detail & Related papers  (2026-06-14T05:40:16Z)
• Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows [18.6534256358905]
  We introduce Harness-Bench, a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent systems.<n>The benchmark contains 106 sandboxed offline tasks constructed from practical agent-use patterns.<n>Across 5,194 execution trajectories, we observe substantial variation in completion, process quality, efficiency, and failure behavior.
  arXiv  Detail & Related papers  (2026-05-27T03:47:35Z)
• RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations [51.43574078961796]
  We introduce RepoMirage, a two-stage evaluation suite built on SWE-Bench Verified.<n>RepoMirage-Perturb applies three types of semantics-preserving repository-level perturbations, revealing a clear performance drop when correct solving requires broader context access.<n>We propose RepoAnchor, a structure-first prototype workflow that separates repository exploration from downstream problem solving.
  arXiv  Detail & Related papers  (2026-05-25T06:26:43Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation [71.49152943451328]
  We introduce a benchmark AJ-Bench to evaluate Agent-as-a-Judge across three domains-search, data systems, and graphical user interfaces-comprising 155 tasks and 516 annotated trajectories.<n>Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification.
  arXiv  Detail & Related papers  (2026-04-20T13:23:38Z)
• Reasoning Provenance for Autonomous AI Agents: Structured Behavioral Analytics Beyond State Checkpoints and Execution Traces [0.0]
  Agent Execution Record (AER) is a structured reasoning primitive that captures intent, observation, and inference as first-class queryable fields on every step.<n>We show how AERs enable population-level behavioral analytics: reasoning pattern mining, confidence calibration, cross-agent comparison, counterfactual regression testing via mock replay.
  arXiv  Detail & Related papers  (2026-03-23T08:27:54Z)
• A Trace-Based Assurance Framework for Agentic AI Orchestration: Contracts, Testing, and Governance [0.22940141855172028]
  This paper presents an assurance framework for Agentic AI systems using Large Language Models (LLMs)<n>Executions are instrumented as Message-Action Traces (MAT) with explicit step and trace contracts.<n>The framework includes stress testing, formulated as a budgeted counterexample search over bounded perturbations.
  arXiv  Detail & Related papers  (2026-03-18T10:23:48Z)
• Evidence-Driven Reasoning for Industrial Maintenance Using Heterogeneous Data [1.5991617440415131]
  Condition Insight Agent integrates maintenance language, behavioral abstractions of operational data, and engineering failure semantics to produce evidence-grounded explanations and advisory actions.<n>System constrains reasoning through deterministic evidence construction and structured failure knowledge, and applies a rule-based verification loop to suppress unsupported conclusions.
  arXiv  Detail & Related papers  (2026-03-09T09:51:50Z)
• Leveraging LLM Parametric Knowledge for Fact Checking without Retrieval [60.25608870901428]
  Trustworthiness is a core research challenge for agentic AI systems built on Large Language Models (LLMs)<n>We propose the task of fact-checking without retrieval, focusing on the verification of arbitrary natural language claims, independent of their source robustness.
  arXiv  Detail & Related papers  (2026-03-05T18:42:51Z)
• Multi-Sourced, Multi-Agent Evidence Retrieval for Fact-Checking [47.47518672198846]
  Misinformation spreading over the Internet poses a significant threat to both societies and individuals.<n>Previous methods rely on semantic and social-contextual patterns learned from training data.<n>We propose WKGFC, which exploits authorized open knowledge graph as a core resource of evidence.
  arXiv  Detail & Related papers  (2026-02-27T19:29:01Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• Transparent, Evaluable, and Accessible Data Agents: A Proof-of-Concept Framework [0.0]
  This article presents a modular, component-based architecture for developing and evaluating AI agents.<n>The system addresses core challenges in data accessibility by enabling non-technical users to interact with complex data warehouses.<n>A cornerstone of the design is its commitment to transparent decision-making, achieved through a multi-layered reasoning framework.
  arXiv  Detail & Related papers  (2025-09-28T23:54:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.