Trace Integrity for LLM Data Agents: A Vision for Auditable Structured Reasoning in Real-World Systems

• URL: http://arxiv.org/abs/2608.26036v1
• Date: Wed, 26 Aug 2026 17:15:24 GMT
• Title: Trace Integrity for LLM Data Agents: A Vision for Auditable Structured Reasoning in Real-World Systems
• Abstract summary: In structured-data tasks, a benchmark-correct answer can be produced by an invalid trace.<n>This paper introduces Trace Integrity, a deployment reliability criterion for evaluating whether the computation recorded behind an answer is explicit.<n>We also introduce CAIT (Correct Answer / Invalid Trace) Rate, which measures how often answer-only evaluation counts computationally unsupported outputs as successes.
• Score: 0.519554837386174
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Answer accuracy is an insufficient reliability signal for LLM data agents. In structured-data tasks, a benchmark-correct answer can be produced by an invalid trace. This paper introduces Trace Integrity, a deployment reliability criterion for evaluating whether the computation recorded behind an answer is explicit, executable, schema-valid, operator-faithful, replayable, answer-consistent, and auditable. We identify the Structure Gap as the deployment failure mode that makes Trace Integrity necessary: natural-language reasoning and free-form rationales do not reliably specify the operator-level programs required by real-world systems. We operationalize Trace Integrity with execution contracts, structured artifacts that bind user intent to schema elements, operator plans, assumptions, executable queries, verification status, and final-answer linkage. We also introduce CAIT (Correct Answer / Invalid Trace) Rate, which measures how often answer-only evaluation counts computationally unsupported outputs as successes. In an empirical demonstration on BIRD Mini-Dev, Direct SQL, Operation Summary + SQL, and Contract-First SQL achieve answer accuracies of 20%, 22%, and 24%, while their Trace Integrity Pass Rates are 39%, 43%, and 40% and their CAIT Rates remain high at 55%, 59.1%, and 45.8%, showing that answer accuracy, trace validity, and silent-failure risk are distinct evaluation signals. Real-world LLM data agents should, therefore, be evaluated not only by whether their outputs match a reference answer, but by whether those outputs are backed by auditable computation.

Related papers

• Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation [46.74484954631231]
  We propose PTA-IRT, a Privileged Trajectory-Aware Item Response Theory framework that fuses process and outcome signals.<n>Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks.
  arXiv  Detail & Related papers  (2026-09-01T17:59:46Z)
• From Sequence to Structure: Relational Uncertainty Propagation for LLM Agents [75.69180947909148]
  We propose a trajectory-level UQ framework for large language model (LLM) agents.<n>RuPA represents an execution history as a directed trajectory graph in which reasoning states, tool interactions, and environment feedback are nodes connected by temporal and semantic dependency edges.<n>We evaluate RUPA on representative agent benchmarks, including $$-2, Terminal-Bench-2, and GAIA, using 6 open-source LLMs spanning multiple model families.
  arXiv  Detail & Related papers  (2026-08-17T01:40:14Z)
• ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents [23.840537431318296]
  We present ClawTrack, a dual-assessment benchmark that simultaneously measures what an agent achieves and how it achieves it.<n>ClawTrack comprises 320 tasks across 8 domains with 25+ deterministic mock services.<n>A Process Grader scores each reasoning turn along four dimensions (goal alignment, efficiency, information utilization, and result verification) anchored by 12,541 task-specific items.
  arXiv  Detail & Related papers  (2026-07-30T11:18:47Z)
• CDR-Bench: Evaluating Faithful Execution of Compositional, Order-Sensitive Data Refinement Recipes [52.734097662607375]
  CDR-Bench is a comprehensive benchmark featuring 3,462 high-quality tasks spanning four real-world data refinement domains.<n>Our benchmark evaluates models across atomic, order-agnostic, and order-sensitive settings.<n>Experiments on 10+ state-of-the-art LLMs reveal consistent failure patterns.
  arXiv  Detail & Related papers  (2026-06-30T10:08:45Z)
• Cross-Vendor Sola ISPM Benchmark: Evaluating Agentic AI for Federated Identity Security Reasoning [0.41942958779358674]
  Existing evaluations focus on isolated single-platform environments.<n>We introduce the Cross-Vendor Sola ISPM Benchmark, a production-grade benchmark of 50 data-grounded tasks.<n>We also contribute an evaluation framework measuring not only final answer correctness but also evidentiary grounding.
  arXiv  Detail & Related papers  (2026-06-01T12:41:00Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis [68.28714988482703]
  Process Reward Models (PRMs) have achieved remarkable success in augmenting the reasoning capabilities of Large Language Models (LLMs)<n>We present a study revealing that general-domain PRMs struggle to supervise data analysis agents.<n>We introduce DataPRM, a novel environment-aware generative process reward model.
  arXiv  Detail & Related papers  (2026-04-27T09:00:30Z)
• Runtime Execution Traces Guided Automated Program Repair with Multi-Agent Debate [8.424102114588559]
  Automated Program Repair (APR) struggles with complex logic errors and silent failures.<n>Current LLM-based APR methods are mostly static, relying on source code and basic test outputs.<n>We propose TraceRepair, a multi-agent framework that leverages runtime facts as shared constraints for patch validation.
  arXiv  Detail & Related papers  (2026-04-03T02:23:25Z)
• Can LLM Agents Generate Real-World Evidence? Evaluating Observational Studies in Medical Databases [17.35673829214932]
  We introduce RWE-bench, a benchmark grounded in MIMIC-IV and derived from peer-reviewed observational studies.<n>Each task provides the corresponding study protocol as the reference standard, requiring agents to execute experiments in a real database.<n>Across 162 tasks, task success is low: the best agent reaches 39.9%, and the best open-source model reaches 30.4%.
  arXiv  Detail & Related papers  (2026-03-24T03:50:34Z)
• VIRO: Robust and Efficient Neuro-Symbolic Reasoning with Verification for Referring Expression Comprehension [51.76841625486355]
  Referring Expression (REC) aims to localize the image region corresponding to a natural-language query.<n>Recent neuro-symbolic REC approaches leverage large language models (LLMs) and vision-language models (VLMs) to perform compositional reasoning.<n>We introduce VIRO, a neuro-symbolic framework that embeds lightweight operator-level verifiers within reasoning steps.
  arXiv  Detail & Related papers  (2026-01-19T07:21:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.