LEDGER: Claim-to-Evidence Trace Graphs for Auditing LLM Agents

• URL: http://arxiv.org/abs/2608.18398v1
• Date: Wed, 19 Aug 2026 00:10:04 GMT
• Title: LEDGER: Claim-to-Evidence Trace Graphs for Auditing LLM Agents
• Abstract summary: LEDGER builds trace graphs over observed agent sessions.<n>We show how the resulting traces expose workflow decisions, artifact lineage, repair steps, validation steps, and claim-support paths for evidence-centered audit.
• Score: 2.9951905451976106
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model (LLM) agents can now carry out long-horizon technical workflows involving complex tool use, code execution, file edits, and generated artifacts. As agents do more work faster, the productivity bottleneck shifts from producing outputs to auditing whether those outputs are correct and trustworthy. Agent observability systems make fine-grained execution events visible, but visibility alone still leaves reviewers to reconstruct which actions, artifacts, and validation steps matter for a particular conclusion. We introduce LEDGER - Layered Evidence and Decision Graphs for Execution Review, a tracing and review system that builds layered trace graphs over observed agent sessions. LEDGER preserves Trace Records while grouping them into Evidence Nodes and Workflow Nodes, representing artifacts as evidence anchors, and adding typed semantic edges that connect claims to supporting actions, artifacts, and checks. Through data-analysis and coding examples, we show how the resulting traces expose workflow decisions, artifact lineage, repair steps, validation coverage, and claim-support paths for evidence-centered audit.

Related papers

• Terminal Agents: A Survey of AI Agents in Command-Line Environments [102.48222742145474]
  Large language model agents increasingly act through terminals, yet existing surveys disperse terminal-mediated behavior across software engineering, tool use, and computer-use research.<n>We regard terminal agents as systems whose dominant progress-bearing action--observation loop is mediated by terminal command execution, textual feedback, and stateful environment interaction.<n>Our synthesis shows that realized behavior is jointly shaped by the model, interface, harness, runtime, and environment.
  arXiv  Detail & Related papers  (2026-08-20T18:16:44Z)
• Formal Verification of Agentic Systems over Operational Data [59.98246281888422]
  Agentic systems driven by large language models (LLMs) are increasingly deployed in real-world where they act on persistent operational data.<n>Existing approaches do not provide such system-level guarantees.<n>We study here the verification of agentic systems comprising a single LLM and a tool orchestration harness over relational operational data.
  arXiv  Detail & Related papers  (2026-08-04T13:01:30Z)
• DocTrace: Towards Traceable Long Document VQA via Hierarchical Evidence Graph Reasoning [14.841209914774778]
  Long Document Visual Question Answering (LongDocVQA) requires Multimodal Large Language Models (MLLMs) to locate, integrate, and reason over heterogeneous document elements distributed across multiple pages.<n>Existing approaches, including end-to-end MLLMs, often lack explicit mechanisms to represent and verify how grounded evidence is progressively composed during reasoning.<n>We propose DocTrace, a hierarchical framework that progressively performs evidence localization, structured document parsing, and evidence graph reasoning.
  arXiv  Detail & Related papers  (2026-08-04T08:04:59Z)
• Execution-First Synthetic Tool-Use Trace Generation for LLM Agents [10.291533584425252]
  SyntheticAgentTraceQA is an execution- first framework for generating scalable supervision data for tool- augmented agents.<n>We evaluate the framework across four tool ecosystems and use the resulting data to fine-tune and evaluate Qwen model variants.
  arXiv  Detail & Related papers  (2026-07-31T08:52:26Z)
• AgentTrails: Towards Trust and Reuse for Agentic Tasks [12.615897924172812]
  LLM-powered agents tackle complex tasks by invoking tools, querying databases, executing code, and manipulating intermediate artifacts.<n>These agents follow trajectories that are typically stored as chronological logs, obscuring the underlying dataflow.<n>We present AgentTrails, a prototype system for agent provenance and sensemaking.
  arXiv  Detail & Related papers  (2026-07-21T07:53:58Z)
• VeriGraph: Towards Verifiable Data-Analytic Agents [71.17693022916896]
  LLM-based agents have demonstrated strong capabilities in data-intensive analytical tasks, yet their outputs are rarely verifiable.<n>We propose VeriGraph, a traceable neuro-symbolic reasoning framework that enables agents to construct an explicit heterogeneous evidence directed acyclic graph.<n>Our results suggest that explicit evidence-graph construction is a promising path toward verifiable data-analytic agents.
  arXiv  Detail & Related papers  (2026-06-15T11:50:56Z)
• Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence [31.954261925882452]
  This survey examines systems that generate, edit, refine, or reason with code under visually grounded inputs and outputs.<n>We first formulate the field by the role that code plays in each task.<n>We then organize benchmarks and methods into four domains: Graphical User Interface, Scientific Visualization, Structured Graphics, and Frontier Tasks and Frameworks.
  arXiv  Detail & Related papers  (2026-06-14T17:21:43Z)
• From Question Answering to Task Completion: A Survey on Agent System and Harness Design [67.8241530947325]
  This survey examines LLM-based agents through a model-harness lens.<n>We first clarify the functional definition of agents and the implementation view of an LLM-based agent as a foundation model and an execution harness.<n>We then analyze the limits of model-centric scaling, trace four paradigms of agent engineering, and decompose the execution harness into six coupled runtime responsibilities.
  arXiv  Detail & Related papers  (2026-06-14T05:40:16Z)
• From Agent Traces to Trust: Evidence Tracing and Execution Provenance in LLM Agents [20.651400118242346]
  Large language model (LLM)-based agents increasingly solve complex tasks by interacting with external tools, retrieval systems, memory modules, environments, and other agents.<n>Final-answer accuracy alone cannot explain how an output was produced, which evidence supported each claim, whether tool calls were justified, how memory influenced later decisions, or where execution failures originated.<n>Evidence tracing and execution provenance address this gap by modeling how retrieved evidence, tool outputs, memory items, environment observations, intermediate claims, actions, and final answers are connected throughout agent execution.
  arXiv  Detail & Related papers  (2026-06-03T15:12:47Z)
• Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows [18.6534256358905]
  We introduce Harness-Bench, a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent systems.<n>The benchmark contains 106 sandboxed offline tasks constructed from practical agent-use patterns.<n>Across 5,194 execution trajectories, we observe substantial variation in completion, process quality, efficiency, and failure behavior.
  arXiv  Detail & Related papers  (2026-05-27T03:47:35Z)
• AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation [71.49152943451328]
  We introduce a benchmark AJ-Bench to evaluate Agent-as-a-Judge across three domains-search, data systems, and graphical user interfaces-comprising 155 tasks and 516 annotated trajectories.<n>Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification.
  arXiv  Detail & Related papers  (2026-04-20T13:23:38Z)
• Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding [49.26132236798123]
  Vision Language Models (VLMs) have gradually become a primary approach in document understanding.<n>We propose SLEUTH, a multi agent framework that orchestrates a retriever and four collaborative agents in a coarse to fine process.<n>The framework identifies key textual and visual clues within the retrieved pages, filters for salient visual evidence such as tables and charts, and analyzes the query to devise a reasoning strategy.
  arXiv  Detail & Related papers  (2025-11-28T03:09:40Z)
• Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers [121.53749383203792]
  We present a holistic end-to-end solution for annotating the factuality of large language models (LLMs)-generated responses. We construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document. Preliminary experiments show that FacTool, FactScore and Perplexity are struggling to identify false claims.
  arXiv  Detail & Related papers  (2023-11-15T14:41:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.