Evaluating Enterprise Analytics Agents: An End-to-End, Trace-Backed Methodology

• URL: http://arxiv.org/abs/2609.09182v1
• Date: Fri, 28 Aug 2026 20:57:45 GMT
• Title: Evaluating Enterprise Analytics Agents: An End-to-End, Trace-Backed Methodology
• Abstract summary: We present an end-to-end evaluation methodology for analytics agents.<n> Grading uses question banks with human-written golden answers, repeated runs, and runtime traces.<n>We instantiate the methodology on a controlled internal analytics agent at a large online marketplace.
• Score: 10.411056951115214
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Enterprise analytics agents are not only text-to-SQL systems. They interpret business intent and choose metric definitions. They select data sources, execute tools, inspect results, and produce natural-language answers. Those answers may influence operational, financial, or executive decisions. Grading final answers hides where these agents fail. A plausible answer can use the wrong source of truth. It can skip a required decomposition, claim causality without support, or change its table interpretation across repeated runs. We present an end-to-end evaluation methodology for analytics agents. The methodology grades agent behavior across three families: semantic understanding, execution quality, and reliability. Grading uses question banks with human-written golden answers, repeated runs, and runtime traces. Each run first passes a run-validity check, then receives tiered, abstention-aware scores that feed a decision framework rather than a release gate. We instantiate the methodology on a controlled internal analytics agent at a large online marketplace. The case study uses 50 analytics questions, two anonymous model configurations, and three randomized repetitions per configuration, yielding 300 traces. The higher-capability configuration reduced early refusal from 73% to 0% and increased real-data answers from 21% to 73%. However, it also exhausted the tool-round budget on 16% of runs. It overran the schema-exploration budget on 77% of traces. It changed its table interpretation on 41 of 50 questions. On finance questions with structured golden answers, source table use and escalation improved, but canonical decomposition remained weak in both configurations. These results show why trust in analytics agents requires end-to-end, trace-backed evaluation rather than SQL correctness or final answer quality alone.

Related papers

• SearchAtlas: Analyzing Agentic Search Strategies via Evidential Query Graphs [30.494397052362384]
  We introduce SearchAtlas, a framework that converts search trajectories into structured graphs.<n>Our automated parsing pipeline achieves a mean edge F1 of 86.4% against human-annotated graphs.<n>We analyze five search agents on three benchmarks, revealing systematic differences in search scale and evidence aggregation.
  arXiv  Detail & Related papers  (2026-09-09T23:09:26Z)
• Governance at the Boundary: How Agent Decomposition Degrades Policy Compliance [6.411104724993454]
  We introduce Fiducia-bench, a benchmark for the governability of financial agents.<n>We use it to study a question no prior benchmark addresses: does an agent degrade its governance?<n>Policy-relevant facts discovered by one component are attenuated at the handoff boundary before reaching the component that must act on them.
  arXiv  Detail & Related papers  (2026-08-17T03:31:59Z)
• Business Truth, not SQL Accuracy: A Rule-Gated 7B Analytics Agent Outperforms a Direct-Prompted 32B Baseline [0.0]
  This paper introduces WarehouseBench, 400 frozen tasks over two synthetic warehouses in which roughly half the correct responses are a clarification, an abstention or a refusal.<n>QueryProof, a 7B agent, uses rules derived from a semantic layer and physical catalog to determine its behaviour.
  arXiv  Detail & Related papers  (2026-08-10T08:14:24Z)
• WorkSurface-Bench: Benchmarking Enterprise Agents on Multi-Surface Knowledge Routing [10.956110208321556]
  We introduce WorkSurface-Bench, a benchmark for evaluating this capability as surface routing.<n>It contains 1,151 atomic tasks derived from persona-scoped Workspace-Bench-Lite workspaces.<n>We evaluate four model backbones across six controlled agent settings, yielding 27,624 protocol-error-free trajectories.
  arXiv  Detail & Related papers  (2026-07-28T14:19:59Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• AgentFinVQA: A Deployable Multi-Agent Pipeline for Auditable Financial Chart QA [7.135926644099901]
  We present AgentFinVQA, a multi-agent pipeline that decomposes each query into planning, OCR, legend grounding, visual inspection, and verification.<n>On FinMME, AgentFinVQA improves $+7.68$ pp over a primary-backbone matched zero-shot baseline with a proprietary backbone.<n>Error analysis shows that question misunderstanding, legend confusion and extraction error account for nearly two-thirds of failures.
  arXiv  Detail & Related papers  (2026-06-18T04:33:07Z)
• ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
  ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
  arXiv  Detail & Related papers  (2026-04-05T17:55:23Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis [0.0]
  TDAD builds a dependency map between source code and tests so that before committing a patch, the agent knows which tests to verify and can self-correct.<n>This paper presents TDAD, an open-source tool that performs pre-change impact analysis for AI coding agents.
  arXiv  Detail & Related papers  (2026-03-18T17:38:22Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Beyond the Last Answer: Your Reasoning Trace Uncovers More than You Think [51.0691253204425]
  We analyze intermediate reasoning steps, termed subthoughts, to answer two questions: Does the final answer reliably represent the model's optimal conclusion?<n>Our approach involves segmenting a reasoning trace into sequential subthoughts based on linguistic cues.<n>We find that aggregating these answers by selecting the most frequent one (the mode) often yields significantly higher accuracy compared to relying solely on the answer derived from the original complete trace.
  arXiv  Detail & Related papers  (2025-04-29T12:39:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.