Fuzzy Integration of Data Lake Tables

• URL: http://arxiv.org/abs/2501.09211v1
• Date: Thu, 16 Jan 2025 00:06:33 GMT
• Title: Fuzzy Integration of Data Lake Tables
• Abstract summary: We present a novel data-driven approach to enable the joining of approximate or fuzzy matches within Full Disjunction.<n>We show that fuzzy Full Disjunction does not add significant time overhead over a state-of-the-art Full Disjunction implementation.
• Score: 9.758870410954271
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Data integration is an important step in any data science pipeline where the objective is to unify the information available in different datasets for comprehensive analysis. Full Disjunction, which is an associative extension of the outer join operator, has been shown to be an effective operator for integrating datasets. It fully preserves and combines the available information. Existing Full Disjunction algorithms only consider the equi-join scenario where only tuples having the same value on joining columns are integrated. This, however, does not realistically represent an open data scenario, where datasets come from diverse sources with inconsistent values (e.g., synonyms, abbreviations, etc.) and with limited metadata. So, joining just on equal values severely limits the ability of Full Disjunction to fully combine datasets. Thus, in this work, we propose an extension of Full Disjunction to also account for "fuzzy" matches among tuples. We present a novel data-driven approach to enable the joining of approximate or fuzzy matches within Full Disjunction. Experimentally, we show that fuzzy Full Disjunction does not add significant time overhead over a state-of-the-art Full Disjunction implementation and also that it enhances the integration effectiveness.

Related papers

• MaDI-Bench: An End-to-End Data Integration Benchmark [3.1427994341585688]
  This paper introduces the Mannheim Data Integration Benchmark (MaDI-Bench)<n>MaDI-Bench is the first benchmark for the end-to-end integration of relational tables covering all steps of the integration process.<n>We validate the benchmark using human-engineered pipelines, a best-of-breed pipeline, and an LLM-based pipeline.
  arXiv  Detail & Related papers  (2026-06-29T14:34:22Z)
• Kernel Representation and Similarity Measure for Incomplete Data [55.62595187178638]
  Measuring similarity between incomplete data is a fundamental challenge in web mining, recommendation systems, and user behavior analysis.<n>Traditional approaches either discard incomplete data or perform imputation as a preprocessing step, leading to information loss and biased similarity estimates.<n>This paper presents a new similarity measure that directly computes similarity between incomplete data in kernel feature space without explicit imputation in the original space.
  arXiv  Detail & Related papers  (2025-10-15T09:41:23Z)
• Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training [53.07879717463279]
  textscDomain2Vec decomposes any dataset into a linear combination of several emphmeta-domains<n>textscDomain2Vec helps find the data mixture that enhances downstream task performance with minimal computational overhead.
  arXiv  Detail & Related papers  (2025-06-12T17:53:51Z)
• Robust Table Integration in Data Lakes [11.719923132819158]
  In this paper, we investigate the challenge of integrating tables from data lakes.<n>We focus on three core tasks: pairwise integrability judgment, integrable set discovery, and multi-tuple conflict resolution.
  arXiv  Detail & Related papers  (2024-11-30T02:45:01Z)
• TabDiff: a Mixed-type Diffusion Model for Tabular Data Generation [91.50296404732902]
  We introduce TabDiff, a joint diffusion framework that models all mixed-type distributions of tabular data in one model. Our key innovation is the development of a joint continuous-time diffusion process for numerical and categorical data. TabDiff achieves superior average performance over existing competitive baselines, with up to $22.5%$ improvement over the state-of-the-art model on pair-wise column correlation estimations.
  arXiv  Detail & Related papers  (2024-10-27T22:58:47Z)
• Anno-incomplete Multi-dataset Detection [67.69438032767613]
  We propose a novel problem as "-incomplete Multi-dataset Detection" We develop an end-to-end multi-task learning architecture which can accurately detect all the object categories with multiple partially annotated datasets.
  arXiv  Detail & Related papers  (2024-08-29T03:58:21Z)
• Finetuning BERT on Partially Annotated NER Corpora [0.0]
  This work presents an approach to finetuning BERT on partially labelled datasets using self-supervision and label preprocessing. Our approach outperforms the previous LSTM-based label preprocessing baseline, significantly improving the performance on poorly labelled datasets.
  arXiv  Detail & Related papers  (2022-11-25T19:54:30Z)
• MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction [78.61546292830081]
  We construct a large-scale human-annotated ERE dataset MAVEN-ERE with improved annotation schemes. It contains 103,193 event coreference chains, 1,216,217 temporal relations, 57,992 causal relations, and 15,841 subevent relations. Experiments show that ERE on MAVEN-ERE is quite challenging, and considering relation interactions with joint learning can improve performances.
  arXiv  Detail & Related papers  (2022-11-14T13:34:49Z)
• ASDOT: Any-Shot Data-to-Text Generation with Pretrained Language Models [82.63962107729994]
  Any-Shot Data-to-Text (ASDOT) is a new approach flexibly applicable to diverse settings. It consists of two steps, data disambiguation and sentence fusion. Experimental results show that ASDOT consistently achieves significant improvement over baselines.
  arXiv  Detail & Related papers  (2022-10-09T19:17:43Z)
• Fusion Subspace Clustering for Incomplete Data [0.0]
  This paper introduces em fusion subspace clustering, a novel method to learn low-dimensional structures that approximate large scale yet highly incomplete data. Our method allows low, high, and even full-rank data; it directly accounts for noise, and its sample complexity approaches the information-theoretic limit. We show through extensive experiments on real and synthetic data that our approach performs comparably to the state-of-the-art with complete data, and dramatically better if data is missing.
  arXiv  Detail & Related papers  (2022-05-22T17:23:41Z)
• Partially-Aligned Data-to-Text Generation with Distant Supervision [69.15410325679635]
  We propose a new generation task called Partially-Aligned Data-to-Text Generation (PADTG) It is more practical since it utilizes automatically annotated data for training and thus considerably expands the application domains. Our framework outperforms all baseline models as well as verify the feasibility of utilizing partially-aligned data.
  arXiv  Detail & Related papers  (2020-10-03T03:18:52Z)
• Self-supervised Robust Object Detectors from Partially Labelled Datasets [3.1669406516464007]
  merging datasets allows us to train one integrated object detector, instead of training several ones. We propose a training framework to overcome missing-label challenge of the merged datasets. We evaluate our proposed framework for training Yolo on a simulated merged dataset with missing rate $approx!48%$ using VOC2012 and VOC2007.
  arXiv  Detail & Related papers  (2020-05-23T15:18:20Z)
• New advances in enumerative biclustering algorithms with online partitioning [80.22629846165306]
  This paper further extends RIn-Close_CVC, a biclustering algorithm capable of performing an efficient, complete, correct and non-redundant enumeration of maximal biclusters with constant values on columns in numerical datasets. The improved algorithm is called RIn-Close_CVC3, keeps those attractive properties of RIn-Close_CVC, and is characterized by: a drastic reduction in memory usage; a consistent gain in runtime.
  arXiv  Detail & Related papers  (2020-03-07T14:54:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.