Exploring the Semantic Gap in Agentic Data Systems: A Formative Study of Operationalization Failures in Analytical Workflows

• URL: http://arxiv.org/abs/2607.00828v1
• Date: Wed, 01 Jul 2026 11:49:50 GMT
• Title: Exploring the Semantic Gap in Agentic Data Systems: A Formative Study of Operationalization Failures in Analytical Workflows
• Abstract summary: We present a cross-domain formative study of operationalization failures in agent-generated analytical concepts.<n>Our analysis reveals five recurring classes of failures: comparative grounding, process reasoning, quantitative reasoning, role confusion, and policy grounding.<n>These findings suggest a semantic gap between user-level analytical concepts and the information available to workflow-generation systems.
• Score: 4.267594643533591
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly used to generate queries, invoke tools, and construct analytical workflows. Although recent advances have substantially improved workflow generation and execution, the semantic information required to operationalize analytical concepts often lies beyond what is explicitly represented in database schemas and data values. We present a cross-domain formative study of operationalization failures in agent-generated analytical workflows. Across 236 analytical intents spanning finance, human resources, and public safety domains, we identify 153 recurring failures despite successful workflow generation and execution. Our analysis reveals five recurring classes of failures: comparative grounding, process reasoning, quantitative reasoning, role confusion, and policy grounding. These findings suggest a semantic gap between user-level analytical concepts and the information available to workflow-generation systems. More broadly, they raise questions about the admissibility of analytical operations and suggest that future agentic data systems may require richer semantic representations to bridge the gap between analytical intent and executable computation.

Related papers

• Pitfalls in Evaluating Interpretability Agents [91.49742416116635]
  We build an agentic system in which a research agent iteratively designs experiments and refines hypotheses.<n>Our work demonstrates fundamental challenges in evaluating complex automated interpretability systems.
  arXiv  Detail & Related papers  (2026-03-20T16:27:17Z)
• CausalAgent: A Conversational Multi-Agent System for End-to-End Causal Inference [36.88497246316067]
  CausalAgent is a conversational multi-agent system for end-to-end causal inference.<n>As a novel user-centered human-AI collaboration paradigm, CausalAgent explicitly models the analysis workflow.
  arXiv  Detail & Related papers  (2026-02-12T03:36:29Z)
• I2I-STRADA -- Information to Insights via Structured Reasoning Agent for Data Analysis [0.0]
  Real-world data analysis requires a consistent cognitive workflow.<n>We introduce I2I-STRADA, an agentic architecture designed to formalize this reasoning process.
  arXiv  Detail & Related papers  (2025-07-23T18:58:42Z)
• Leveraging Knowledge Graphs and LLM Reasoning to Identify Operational Bottlenecks for Warehouse Planning Assistance [1.2749527861829046]
  Our framework integrates Knowledge Graphs (KGs) and Large Language Model (LLM)-based agents.<n>It transforms raw DES data into a semantically rich KG, capturing relationships between simulation events and entities.<n>An LLM-based agent uses iterative reasoning, generating interdependent sub-questions. For each sub-question, it creates Cypher queries for KG interaction, extracts information, and self-reflects to correct errors.
  arXiv  Detail & Related papers  (2025-07-23T07:18:55Z)
• Agentic Predictor: Performance Prediction for Agentic Workflows via Multi-View Encoding [56.565200973244146]
  Agentic Predictor is a lightweight predictor for efficient agentic workflow evaluation.<n>By learning to approximate task success rates, Agentic Predictor enables fast and accurate selection of optimal agentic workflow configurations.
  arXiv  Detail & Related papers  (2025-05-26T09:46:50Z)
• Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning [50.53703102032562]
  Large-scale Transformer language models (LMs) trained solely on next-token prediction with web-scale data can solve a wide range of tasks.<n>The mechanism behind this capability, known as in-context learning (ICL), remains both controversial and poorly understood.
  arXiv  Detail & Related papers  (2025-05-16T08:50:42Z)
• TRAIL: Trace Reasoning and Agentic Issue Localization [5.025960714013197]
  This work articulates the need for robust and dynamic evaluation methods for agentic workflow traces.<n>We present a set of 148 large human-annotated traces (TRAIL) constructed using this taxonomy and grounded in established agentic benchmarks.<n>To ensure ecological validity, we curate traces from both single and multi-agent systems.
  arXiv  Detail & Related papers  (2025-05-13T14:55:31Z)
• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures.<n>We also present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms.<n>We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)
• Data Analysis in the Era of Generative AI [56.44807642944589]
  This paper explores the potential of AI-powered tools to reshape data analysis, focusing on design considerations and challenges. We explore how the emergence of large language and multimodal models offers new opportunities to enhance various stages of data analysis workflow. We then examine human-centered design principles that facilitate intuitive interactions, build user trust, and streamline the AI-assisted analysis workflow across multiple apps.
  arXiv  Detail & Related papers  (2024-09-27T06:31:03Z)
• A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis [128.0532113800092]
  We present a mechanistic interpretation of Transformer-based LMs on arithmetic questions. This provides insights into how information related to arithmetic is processed by LMs.
  arXiv  Detail & Related papers  (2023-05-24T11:43:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.