An Agentic Retrieval Framework for Autonomous Context-Aware Data Quality Assessment

• URL: http://arxiv.org/abs/2606.13692v1
• Date: Fri, 15 May 2026 11:24:53 GMT
• Title: An Agentic Retrieval Framework for Autonomous Context-Aware Data Quality Assessment
• Abstract summary: We propose a unified agentic-retrieval framework for autonomous context-aware data quality assessment.<n>The framework interprets natural-language descriptions of intended data usage, derives context-aware assessment strategies, and generates executable validation logic.<n>We implement the proposed framework as an end-to-end prototype and evaluate it across multiple usage scenarios applied to the same dataset.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Data quality assessment is a critical prerequisite for effective data analytics and data-driven decision-making, yet it remains a challenging task due to the inherently context-dependent nature of data quality. Existing approaches often rely on static rules or manual assessment strategies, limiting their adaptability to diverse usage scenarios and constraining automation at scale. Recent advances in artificial intelligence, particularly large language models, offer new opportunities for automating data quality assessment, but raise concerns related to reliability, grounding, and execution safety. In this paper, we propose a unified agentic-retrieval framework for autonomous context-aware data quality assessment. The framework interprets natural-language descriptions of intended data usage, derives context-aware assessment strategies, and generates executable validation logic through a multi-agent workflow. To ensure operational reliability, the framework introduces a feasibility validation stage that evaluates the realism and executability of generated assessment specifications before execution, enabling iterative refinement when necessary. Accepted validation logic is executed deterministically to guarantee reproducible and auditable results. We implement the proposed framework as an end-to-end prototype and evaluate it across multiple usage scenarios applied to the same dataset. The results demonstrate that assessment outcomes adapt meaningfully to different intended uses, while feasibility-gated execution reduces unrealistic or non-executable rule generation. The proposed approach provides a practical foundation for deploying autonomous yet controlled data quality assessment in modern data-driven environments.

Related papers

• A Framework for Generating Valid Context-Specific Benchmarks through Expert Guidance [8.655727081754241]
  This paper presents an end-to-end approach for generating context-specific large language model (LLM) benchmark datasets.<n>We introduce a schema eliciting key information about the goals, scope, and context of an evaluation task, and use this information to guide synthetic data generation.<n>We show how expert-informed scaffolds can guide synthetic data generation toward more valid benchmarks.
  arXiv  Detail & Related papers  (2026-09-15T03:36:35Z)
• Model Validation of Agentic AI Systems: A POMDP-Based Framework for Belief-State, Forecast, and Policy Validation [0.0]
  This paper proposes a model validation framework for agentic AI based on Partially Observable Markov Decision Processes (POMDPs)<n>The framework decomposes autonomous decision making into information, beliefs, forecasts, actions, and utility, allowing each component to be validated independently.<n>The principal contribution of the paper is a practical framework for extending established model risk management concepts to autonomous AI systems.
  arXiv  Detail & Related papers  (2026-06-16T00:40:55Z)
• Evaluating Supervised Machine Learning Models: Principles, Pitfalls, and Metric Selection [5.015895904388096]
  This paper examines the principles, challenges, and practical considerations involved in evaluating supervised learning algorithms.<n>It discusses how evaluation outcomes are influenced by dataset characteristics, validation design, class imbalance, asymmetric error costs, and the choice of performance metrics.
  arXiv  Detail & Related papers  (2026-04-15T13:44:35Z)
• Statistical Confidence in Functional Correctness: An Approach for AI Product Functional Correctness Evaluation [1.4521584395164622]
  This paper proposes and evaluates the Statistical Confidence in Functional Correctness (SCFC) approach.<n>The approach consists of four steps: defining quantitative specification limits, performing stratified and probabilistic sampling, applying bootstrapping to estimate a confidence interval for the performance metric, and calculating a capability index as a final indicator.<n>We conclude that the proposed approach is a feasible and valuable way to operationalize the assessment of functional correctness, moving the evaluation from a point estimate to a statement of statistical confidence.
  arXiv  Detail & Related papers  (2026-02-20T17:06:38Z)
• EffiEval: Efficient and Generalizable Model Evaluation via Capability Coverage Maximization [48.27039405295434]
  EffiEval is a training-free approach for efficient benchmarking that addresses data redundancy while maintaining high evaluation reliability.<n>Our method is specifically designed to meet three key criteria for high-quality evaluation: representativeness, fairness, and generalizability.<n>EffiEval achieves strong ranking consistency with full-dataset evaluation using only a small fraction of the original data.
  arXiv  Detail & Related papers  (2025-08-13T09:48:23Z)
• Cost-Optimal Active AI Model Evaluation [71.2069549142394]
  Development of generative AI systems requires continual evaluation, data acquisition, and annotation.<n>We develop novel, cost-aware methods for actively balancing the use of a cheap, but often inaccurate, weak rater.<n>We derive a family of cost-optimal policies for allocating a given annotation budget between weak and strong raters.
  arXiv  Detail & Related papers  (2025-06-09T17:14:41Z)
• Towards Explainable Automated Data Quality Enhancement without Domain Knowledge [0.0]
  We propose a comprehensive framework designed to automatically assess and rectify data quality issues in any given dataset. Our primary objective is to address three fundamental types of defects: absence, redundancy, and incoherence. We adopt a hybrid approach that integrates statistical methods with machine learning algorithms.
  arXiv  Detail & Related papers  (2024-09-16T10:08:05Z)
• Adaptive Data Quality Scoring Operations Framework using Drift-Aware Mechanism for Industrial Applications [0.0]
  We introduce a novel framework to address the challenges posed by dynamic quality dimensions in industrial data streams. The framework integrates a dynamic change detector mechanism that actively monitors and adapts to changes in data quality. The experimental results reveal high predictive performance and efficient processing time, highlighting its effectiveness in practical quality-driven AI applications.
  arXiv  Detail & Related papers  (2024-08-13T08:32:06Z)
• QualEval: Qualitative Evaluation for Model Improvement [82.73561470966658]
  We propose QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement. QualEval uses a powerful LLM reasoner and our novel flexible linear programming solver to generate human-readable insights. We demonstrate that leveraging its insights, for example, improves the absolute performance of the Llama 2 model by up to 15% points relative.
  arXiv  Detail & Related papers  (2023-11-06T00:21:44Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)
• Position: AI Evaluation Should Learn from How We Test Humans [65.36614996495983]
  We argue that psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution to the challenges in today's AI evaluations.
  arXiv  Detail & Related papers  (2023-06-18T09:54:33Z)
• Approaching Neural Network Uncertainty Realism [53.308409014122816]
  Quantifying or at least upper-bounding uncertainties is vital for safety-critical systems such as autonomous vehicles. We evaluate uncertainty realism -- a strict quality criterion -- with a Mahalanobis distance-based statistical test. We adopt it to the automotive domain and show that it significantly improves uncertainty realism compared to a plain encoder-decoder model.
  arXiv  Detail & Related papers  (2021-01-08T11:56:12Z)
• Causal Feature Selection for Algorithmic Fairness [61.767399505764736]
  We consider fairness in the integration component of data management. We propose an approach to identify a sub-collection of features that ensure the fairness of the dataset.
  arXiv  Detail & Related papers  (2020-06-10T20:20:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.