MaDI-Bench: An End-to-End Data Integration Benchmark

• URL: http://arxiv.org/abs/2606.30371v1
• Date: Mon, 29 Jun 2026 14:34:22 GMT
• Title: MaDI-Bench: An End-to-End Data Integration Benchmark
• Abstract summary: This paper introduces the Mannheim Data Integration Benchmark (MaDI-Bench)<n>MaDI-Bench is the first benchmark for the end-to-end integration of relational tables covering all steps of the integration process.<n>We validate the benchmark using human-engineered pipelines, a best-of-breed pipeline, and an LLM-based pipeline.
• Score: 3.1427994341585688
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Data integration combines heterogeneous data sets into a single, coherent representation. Data integration involves a sequence of interdependent tasks including schema matching, value normalization, entity blocking, entity matching, and data fusion. Existing benchmarks either evaluate these steps in isolation or cover only incomplete versions of the data integration pipeline, omitting specific steps. The lack of public end-to-end data integration benchmarks hinders research on data integration methods that address the integration process as a whole. This paper fills this gap by introducing the Mannheim Data Integration Benchmark (MaDI-Bench), the first benchmark for the end-to-end integration of relational tables covering all steps of the integration process. MaDI-Bench contributes (i) a set of base end-to-end data integration tasks spanning several application domains, each requiring the full schema matching, value normalization, entity matching, and conflict resolution pipeline; and (ii) a generic method for deriving task variants that mitigates rapid benchmark saturation as data integration systems advance. We validate the benchmark using human-engineered pipelines, a best-of-breed pipeline, and an LLM-based pipeline. The validation demonstrates the utility of the benchmark for measuring the step-wise as well as the end-to-end performance of data integration pipelines. All benchmark artifacts are available for public download.

Related papers

• Environment-free Synthetic Data Generation for API-Calling Agents [68.95520739708304]
  Training API-calling large language model (LLM) agents demands massive amounts of high-quality trajectories.<n>We propose an environment-free synthetic data generation approach that leverages LLMs as on-the-fly digital world models.<n>We evaluate our approach on the challenging AppWorld and OfficeBench benchmarks, which include both information-retrieval and state-changing tasks.
  arXiv  Detail & Related papers  (2026-07-18T17:36:44Z)
• BDIViz in Action: Interactive Curation and Benchmarking for Schema Matching Methods [12.444984647326024]
  BDIViz is an interactive visualization system for schema matching with LLM-assisted validation.<n>New extension to BDIViz addresses a critical need in data integration research: human-in-the-loop benchmarking and iterative matcher development.
  arXiv  Detail & Related papers  (2026-04-12T18:16:30Z)
• CUBE: A Standard for Unifying Agent Benchmarks [138.9831263189749]
  We propose CUBE (Common Unified Benchmark Environments), a universal protocol standard built on MCP and Gym.<n>CUBE enables any compliant platform to access any compliant benchmark for evaluation, RL training, or data generation without custom integration.
  arXiv  Detail & Related papers  (2026-03-16T18:31:37Z)
• Automatic End-to-End Data Integration using Large Language Models [3.188426002064626]
  We present an automatic data integration pipeline that uses GPT-5.2 to generate all artifacts required to adapt the pipeline to specific use cases.<n>We compare the performance of this LLM-based pipeline to the performance of human-designed pipelines along three case studies.<n>Our experiments show that the LLM-based pipeline is able to produce similar results, for some tasks even better results, as the human-designed pipelines.
  arXiv  Detail & Related papers  (2026-03-11T08:56:55Z)
• Nimbus: A Unified Embodied Synthetic Data Generation Framework [51.55989844555466]
  Scaling data volume and diversity is critical for generalizing embodied intelligence.<n>We present Nimbus, a unified synthetic data generation framework designed to integrate heterogeneous navigation and manipulation pipelines.<n>Our evaluation demonstrates that Nimbus achieves a 2-3X improvement in end-to-end throughput compared to unoptimized baselines.
  arXiv  Detail & Related papers  (2026-01-29T09:27:31Z)
• KGpipe: Generation and Evaluation of Pipelines for Data Integration into Knowledge Graphs [1.5397834466394758]
  Building high-quality knowledge graphs (KGs) from diverse sources requires combining methods for information extraction, data transformation, ontology mapping, entity matching, and data fusion.<n>We present a new framework, KGpipe for defining and executing pipelines that can combine existing tools or LLM (Large Language Model) functionality.
  arXiv  Detail & Related papers  (2025-11-23T09:21:14Z)
• WikiDBGraph: A Data Management Benchmark Suite for Collaborative Learning over Database Silos [48.88393315169039]
  Collaborative learning (CL) techniques enable multiple parties to train models jointly without sharing raw data.<n>Current CL benchmarks and algorithms primarily target the learning step under assumptions of isolated, aligned, and joinable databases.<n>We build a large-scale dataset constructed from 100,000 real-world relational databases linked by 17 million weighted edges.
  arXiv  Detail & Related papers  (2025-05-22T13:07:06Z)
• Fuzzy Integration of Data Lake Tables [9.758870410954271]
  We present a novel data-driven approach to enable the joining of approximate or fuzzy matches within Full Disjunction.<n>We show that fuzzy Full Disjunction does not add significant time overhead over a state-of-the-art Full Disjunction implementation.
  arXiv  Detail & Related papers  (2025-01-16T00:06:33Z)
• Matchmaker: Self-Improving Large Language Model Programs for Schema Matching [60.23571456538149]
  We propose a compositional language model program for schema matching, comprised of candidate generation, refinement and confidence scoring. Matchmaker self-improves in a zero-shot manner without the need for labeled demonstrations. Empirically, we demonstrate on real-world medical schema matching benchmarks that Matchmaker outperforms previous ML-based approaches.
  arXiv  Detail & Related papers  (2024-10-31T16:34:03Z)
• Interpetable Target-Feature Aggregation for Multi-Task Learning based on Bias-Variance Analysis [53.38518232934096]
  Multi-task learning (MTL) is a powerful machine learning paradigm designed to leverage shared knowledge across tasks to improve generalization and performance. We propose an MTL approach at the intersection between task clustering and feature transformation based on a two-phase iterative aggregation of targets and features. In both phases, a key aspect is to preserve the interpretability of the reduced targets and features through the aggregation with the mean, which is motivated by applications to Earth science.
  arXiv  Detail & Related papers  (2024-06-12T08:30:16Z)
• A sequence-to-sequence approach for document-level relation extraction [4.906513405712846]
  Document-level relation extraction (DocRE) requires integrating information within and across sentences. Seq2rel can learn the subtasks of DocRE end-to-end, replacing a pipeline of task-specific components.
  arXiv  Detail & Related papers  (2022-04-03T16:03:19Z)
• Bellamy: Reusing Performance Models for Distributed Dataflow Jobs Across Contexts [52.9168275057997]
  This paper presents Bellamy, a novel modeling approach that combines scale-outs, dataset sizes, and runtimes with additional descriptive properties of a dataflow job. We evaluate our approach on two publicly available datasets consisting of execution data from various dataflow jobs carried out in different environments.
  arXiv  Detail & Related papers  (2021-07-29T11:57:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.