Table Integration in Data Lakes Unleashed: Pairwise Integrability Judgment, Integrable Set Discovery, and Multi-Tuple Conflict Resolution

• URL: http://arxiv.org/abs/2412.00324v2
• Date: Sun, 13 Apr 2025 13:19:21 GMT
• Title: Table Integration in Data Lakes Unleashed: Pairwise Integrability Judgment, Integrable Set Discovery, and Multi-Tuple Conflict Resolution
• Abstract summary: We train a binary-supervised adversarial learning algorithm to address the task of pairwise integrability judgment.<n>We introduce an innovative in-context learning methodology to tackle multi-tuple conflict resolution.
• Score: 11.719923132819158
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Table integration aims to create a comprehensive table by consolidating tuples containing relevant information. In this work, we investigate the challenge of integrating multiple tables from a data lake, focusing on three core tasks: 1) pairwise integrability judgment, which determines whether a tuple pair is integrable, accounting for any occurrences of semantic equivalence or typographical errors; 2) integrable set discovery, which identifies all integrable sets in a table based on pairwise integrability judgments established in the first task; 3) multi-tuple conflict resolution, which resolves conflicts between multiple tuples during integration. To this end, we train a binary classifier to address the task of pairwise integrability judgment. Given the scarcity of labeled data in data lakes, we propose a self-supervised adversarial contrastive learning algorithm to perform classification, which incorporates data augmentation methods and adversarial examples to autonomously generate new training data. Upon the output of pairwise integrability judgment, each integrable set can be considered as a community, a densely connected sub-graph where nodes and edges correspond to tuples in the table and their pairwise integrability respectively, we proceed to investigate various community detection algorithms to address the integrable set discovery objective. Moving forward to tackle multi-tuple conflict resolution, we introduce an innovative in-context learning methodology. This approach capitalizes on the knowledge embedded within large language models (LLMs) to effectively resolve conflicts that arise when integrating multiple tuples. Notably, our method minimizes the need for annotated data, making it particularly suited for scenarios where labeled datasets are scarce.

Related papers

• End-to-End Verifiable and Robust Federated Learning [0.9134244356393665]
  Federated learning enables multiple parties to train a shared model without centralizing raw data with the help of an aggregator.<n>Two requirements are particularly important: robustness to poisoned or Byzantine client updates, and verifiability of the aggregator.<n>We present a verifiable federated learning protocol that makes a robust aggregation pipeline publicly auditable.
  arXiv  Detail & Related papers  (2026-09-14T13:13:25Z)
• BrokenBind: Universal Modality Exploration beyond Dataset Boundaries [112.81381711545043]
  We introduce BrokenBind, which focuses on binding modalities that are presented from different datasets.<n>Under our framework, any two modalities can be bound together, free from the dataset limitation.
  arXiv  Detail & Related papers  (2026-02-06T07:26:49Z)
• Full Triple Matcher: Integrating all triple elements between heterogeneous Knowledge Graphs [0.09471093245585005]
  Knowledge graphs (KGs) are powerful tools for representing and reasoning over structured information.<n>Current approaches may fall short in scenarios where diverse and complex contexts need to be integrated.<n>We propose a novel KG integration method consisting of label matching and triple matching.
  arXiv  Detail & Related papers  (2025-07-20T07:46:55Z)
• Maximal Matching Matters: Preventing Representation Collapse for Robust Cross-Modal Retrieval [0.5999777817331317]
  Cross-modal image-text retrieval is challenging because of the diverse possible associations between content from different modalities.<n>Traditional methods learn a single-vector embedding to represent semantics of each sample.<n>Set-based approaches, which represent each sample with multiple embeddings, offer a promising alternative.
  arXiv  Detail & Related papers  (2025-06-26T17:55:34Z)
• Fuzzy Integration of Data Lake Tables [9.758870410954271]
  We present a novel data-driven approach to enable the joining of approximate or fuzzy matches within Full Disjunction. We show that fuzzy Full Disjunction does not add significant time overhead over a state-of-the-art Full Disjunction implementation.
  arXiv  Detail & Related papers  (2025-01-16T00:06:33Z)
• MISFEAT: Feature Selection for Subgroups with Systematic Missing Data [8.063972429611365]
  We address the challenge of systematic missing data, a scenario in which some feature values are missing for alls of a subgroup. Our goal is to identify top-K feature subsets of some fixed size with the highest joint mutual information with a target variable. We propose a generalizable model based on heterogeneous graph neural network to identify interdependencies between feature-subgroup-target variable connections.
  arXiv  Detail & Related papers  (2024-12-09T17:59:59Z)
• Prompt-Matcher: Leveraging Large Models to Reduce Uncertainty in Schema Matching Results [1.13107643869251]
  We introduce a new approach based on fine-grained correspondence verification with specific prompt of Large Language Model. Our approach is an iterative loop that consists of three main components: (1) the correspondence selection algorithm, (2) correspondence verification, and (3) the update of probability distribution. We propose a novel $(1-1/e)$-approximation algorithm that significantly outperforms brute algorithm in terms of computational efficiency.
  arXiv  Detail & Related papers  (2024-08-24T16:54:08Z)
• A Bi-consolidating Model for Joint Relational Triple Extraction [3.972061685570092]
  Current methods to extract relational triples directly make a prediction based on a possible entity pair in a raw sentence without depending on entity recognition. The task suffers from a serious semantic overlapping problem, in which several relation triples may share one or two entities in a sentence. A bi-consolidating model is proposed to address this problem by simultaneously reinforcing the local and global semantic features relevant to a relation triple.
  arXiv  Detail & Related papers  (2024-04-05T04:04:23Z)
• Enhancing Neural Subset Selection: Integrating Background Information into Set Representations [53.15923939406772]
  We show that when the target value is conditioned on both the input set and subset, it is essential to incorporate an textitinvariant sufficient statistic of the superset into the subset of interest. This ensures that the output value remains invariant to permutations of the subset and its corresponding superset, enabling identification of the specific superset from which the subset originated.
  arXiv  Detail & Related papers  (2024-02-05T16:09:35Z)
• The Battleship Approach to the Low Resource Entity Matching Problem [0.0]
  We propose a new active learning approach for entity matching problems. We focus on a selection mechanism that exploits unique properties of entity matching. An experimental analysis shows that the proposed algorithm outperforms state-of-the-art active learning solutions to low resource entity matching.
  arXiv  Detail & Related papers  (2023-11-27T10:18:17Z)
• Multi-Grained Multimodal Interaction Network for Entity Linking [65.30260033700338]
  Multimodal entity linking task aims at resolving ambiguous mentions to a multimodal knowledge graph. We propose a novel Multi-GraIned Multimodal InteraCtion Network $textbf(MIMIC)$ framework for solving the MEL task.
  arXiv  Detail & Related papers  (2023-07-19T02:11:19Z)
• Learning Unseen Modality Interaction [54.23533023883659]
  Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved.
  arXiv  Detail & Related papers  (2023-06-22T10:53:10Z)
• Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications [90.6849884683226]
  We study the challenge of interaction quantification in a semi-supervised setting with only labeled unimodal data. Using a precise information-theoretic definition of interactions, our key contribution is the derivation of lower and upper bounds. We show how these theoretical results can be used to estimate multimodal model performance, guide data collection, and select appropriate multimodal models for various tasks.
  arXiv  Detail & Related papers  (2023-06-07T15:44:53Z)
• Towards Integration of Discriminability and Robustness for Document-Level Relation Extraction [41.51148745387936]
  Document-level relation extraction (DocRE) predicts relations for entity pairs that rely on long-range context-dependent reasoning in a document. In this work, we aim to achieve better integration of both the discriminability and robustness for the DocRE problem. We innovatively customize entropy minimization and supervised contrastive learning for the challenging multi-label and long-tailed learning problems.
  arXiv  Detail & Related papers  (2023-04-03T09:11:18Z)
• Deep Learning to Jointly Schema Match, Impute, and Transform Databases [19.200830026362425]
  Joining data from multiple origins with unmapped and only partially overlapping features is a prerequisite to developing robust, generalizable algorithms. We develop two novel procedures to address this problem. In synthetic and real-world experiments using two electronic health record databases, our algorithms outperform existing baselines for matching variable sets.
  arXiv  Detail & Related papers  (2022-06-22T21:25:59Z)
• On Modality Bias Recognition and Reduction [70.69194431713825]
  We study the modality bias problem in the context of multi-modal classification. We propose a plug-and-play loss function method, whereby the feature space for each label is adaptively learned. Our method yields remarkable performance improvements compared with the baselines.
  arXiv  Detail & Related papers  (2022-02-25T13:47:09Z)
• Leveraging Ensembles and Self-Supervised Learning for Fully-Unsupervised Person Re-Identification and Text Authorship Attribution [77.85461690214551]
  Learning from fully-unlabeled data is challenging in Multimedia Forensics problems, such as Person Re-Identification and Text Authorship Attribution. Recent self-supervised learning methods have shown to be effective when dealing with fully-unlabeled data in cases where the underlying classes have significant semantic differences. We propose a strategy to tackle Person Re-Identification and Text Authorship Attribution by enabling learning from unlabeled data even when samples from different classes are not prominently diverse.
  arXiv  Detail & Related papers  (2022-02-07T13:08:11Z)
• Wisdom of the Ensemble: Improving Consistency of Deep Learning Models [11.230300336108018]
  Trust is often a function of constant behavior. This paper studies a model behavior in the context of periodic retraining of deployed models. We prove that consistency and correct-consistency of an ensemble learner is not less than the average consistency and correct-consistency of individual learners.
  arXiv  Detail & Related papers  (2020-11-13T07:47:01Z)
• CoADNet: Collaborative Aggregation-and-Distribution Networks for Co-Salient Object Detection [91.91911418421086]
  Co-Salient Object Detection (CoSOD) aims at discovering salient objects that repeatedly appear in a given query group containing two or more relevant images. One challenging issue is how to effectively capture co-saliency cues by modeling and exploiting inter-image relationships. We present an end-to-end collaborative aggregation-and-distribution network (CoADNet) to capture both salient and repetitive visual patterns from multiple images.
  arXiv  Detail & Related papers  (2020-11-10T04:28:11Z)
• Learning while Respecting Privacy and Robustness to Distributional Uncertainties and Adversarial Data [66.78671826743884]
  The distributionally robust optimization framework is considered for training a parametric model. The objective is to endow the trained model with robustness against adversarially manipulated input data. Proposed algorithms offer robustness with little overhead.
  arXiv  Detail & Related papers  (2020-07-07T18:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.