SANA: What Matters for QA Agents over Massive Data Lakes?

• URL: http://arxiv.org/abs/2606.13904v1
• Date: Thu, 11 Jun 2026 20:51:44 GMT
• Title: SANA: What Matters for QA Agents over Massive Data Lakes?
• Abstract summary: We present SANA, a framework that transforms EQA tasks into runtime profiles containing gold source sequence, sanitized subquestions, and execution records.<n>We evaluate two recent EQA benchmarks, LakeQA and KramaBench, and evaluate lightweight and mid-sized agents under fixed prompts, budgets, data lakes, and runtimes.
• Score: 21.461240255601684
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Exploratory question answering (EQA) over data lakes requires an LLM agent to discover relevant sources, analyze retrieved data, and adapt its actions based on intermediate results. End-to-end accuracy alone cannot distinguish failures in search, planning, data analysis, or the agent's Action Policy: its decisions about what to do next and when to submit an answer. We present SANA (Search Agent Navigation Ablation framework), a diagnostic ablation framework that transforms EQA tasks into runtime profiles containing gold source sequence, sanitized subquestions, and execution records. SANA uses these profiles to construct idealized search, planning, and data-analysis tools, allowing each component to be ablated; the residual gap is diagnostic evidence for policy failures. To illustrate SANA as a reusable evaluation framework, we adapted two recent EQA benchmarks, LakeQA and KramaBench, and evaluated lightweight and mid-sized agents under fixed prompts, budgets, data lakes, and runtimes. Across both benchmarks, data analysis is a consistent bottleneck while planning is less so. Search is a major limitation in LakeQA's large data-lake setting, but less so for the smaller-scale KramaBench. SANA thus deconstructs end-to-end task accuracies into a diagnosis of where data-lake agents fail, and allows for systematic comparisons of progress in search, planning, data analysis, and agent design.

Related papers

• Walk Before You Run: The Importance of Data Exploration for Data Analysis Agents [7.140798105546584]
  In current tools and benchmarks, this step is usually left implicit, creating a gap between downstream task performance and dataset understanding.<n>We show through downstream experiments that stronger Data Exploration support improves task performance.<n>Our results show that strong LLMs and data-analysis agents still miss important logical structure even when they read spreadsheet content.
  arXiv  Detail & Related papers  (2026-08-17T03:19:59Z)
• Unsupervised Skill Discovery for Agentic Data Analysis [55.42853694128929]
  DataCOPE is an unsupervised verifier-guided skill discovery framework for data-analytic agents.<n>It iteratively coordinates a Data-Analytic Agent for trajectory generation, an Unsupervised Verifier for signal extraction, and a Skill Manager for contrastive skill distillation.<n>We evaluate DataCOPE on report-style analysis from Deep Data Research and reasoning-style analysis from DABStep.
  arXiv  Detail & Related papers  (2026-06-04T17:20:47Z)
• "Skill issues'': data-centric optimization of lakehouse agents [2.8558664643005525]
  We show how to optimize artifacts for agents operating on a branching lakehouse, Bauplan.<n>We present a data-centric optimization pipeline that generates task-verifier pairs, executes candidate skills in isolated sandboxes, and scores trajectories using both trace-level signals and programmatic checks over lakehouse state.
  arXiv  Detail & Related papers  (2026-05-31T11:58:04Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• TSAQA: Time Series Analysis Question And Answering Benchmark [85.35545785252309]
  Time series data are integral to critical applications across domains such as finance, healthcare, transportation, and environmental science.<n>We introduce TSAQA, a novel unified benchmark designed to broaden task coverage and evaluate diverse temporal analysis capabilities.
  arXiv  Detail & Related papers  (2026-01-30T17:28:56Z)
• LongDA: Benchmarking LLM Agents for Long-Document Data Analysis [55.32211515932351]
  LongDA targets real-world settings in which navigating long documentation and complex data is the primary bottleneck.<n>LongTA is a tool-augmented agent framework that enables document access, retrieval, and code execution.<n>Our experiments reveal substantial performance gaps even among state-of-the-art models.
  arXiv  Detail & Related papers  (2026-01-05T23:23:16Z)
• GDS Agent for Graph Algorithmic Reasoning [0.0]
  We introduce the GDS (Graph Data Science) agent in this technical report.<n>The GDS agent introduces a comprehensive set of graph algorithms as tools, together with preprocessing (retrieval) and postprocessing of algorithm results.<n>Results indicate that GDS agent is able to solve a wide spectrum of graph tasks.
  arXiv  Detail & Related papers  (2025-08-28T10:35:44Z)
• Leveraging Knowledge Graphs and LLM Reasoning to Identify Operational Bottlenecks for Warehouse Planning Assistance [1.2749527861829046]
  Our framework integrates Knowledge Graphs (KGs) and Large Language Model (LLM)-based agents.<n>It transforms raw DES data into a semantically rich KG, capturing relationships between simulation events and entities.<n>An LLM-based agent uses iterative reasoning, generating interdependent sub-questions. For each sub-question, it creates Cypher queries for KG interaction, extracts information, and self-reflects to correct errors.
  arXiv  Detail & Related papers  (2025-07-23T07:18:55Z)
• DABstep: Data Agent Benchmark for Multi-step Reasoning [2.6709582216950767]
  DABstep is a novel benchmark for evaluating AI agents on realistic multi-step data analysis tasks.<n>It comprises over 450 real-world challenges derived from a financial analytics platform.<n>Dabstep is released with a public leaderboard and toolkit to accelerate research in autonomous data analysis.
  arXiv  Detail & Related papers  (2025-06-30T10:49:21Z)
• InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks [84.7788065721689]
  In this paper, we introduce InfiAgent-DABench, the first benchmark specifically designed to evaluate LLM-based agents on data analysis tasks. This benchmark contains DAEval, a dataset consisting of 257 data analysis questions derived from 52 CSV files. Building on top of our agent framework, we develop a specialized agent, DAAgent, which surpasses GPT-3.5 by 3.9% on DABench.
  arXiv  Detail & Related papers  (2024-01-10T19:04:00Z)
• MinPrompt: Graph-based Minimal Prompt Data Augmentation for Few-shot Question Answering [64.6741991162092]
  We present MinPrompt, a minimal data augmentation framework for open-domain question answering. We transform the raw text into a graph structure to build connections between different factual sentences. We then apply graph algorithms to identify the minimal set of sentences needed to cover the most information in the raw text. We generate QA pairs based on the identified sentence subset and train the model on the selected sentences to obtain the final model.
  arXiv  Detail & Related papers  (2023-10-08T04:44:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.