Heterogeneous Contrastive Learning
- URL: http://arxiv.org/abs/2105.09401v1
- Date: Wed, 19 May 2021 21:01:41 GMT
- Title: Heterogeneous Contrastive Learning
- Authors: Lecheng Zheng, Yada Zhu, Jingrui He, and Jinjun Xiong
- Abstract summary: We propose a unified heterogeneous learning framework, which combines weighted unsupervised contrastive loss and weighted supervised contrastive loss.
Experimental results on real-world data sets demonstrate the effectiveness and the efficiency of the proposed method.
- Score: 45.93509060683946
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: With the advent of big data across multiple high-impact applications, we are
often facing the challenge of complex heterogeneity. The newly collected data
usually consist of multiple modalities and characterized with multiple labels,
thus exhibiting the co-existence of multiple types of heterogeneity. Although
state-of-the-art techniques are good at modeling the complex heterogeneity with
sufficient label information, such label information can be quite expensive to
obtain in real applications, leading to sub-optimal performance using these
techniques. Inspired by the capability of contrastive learning to utilize rich
unlabeled data for improving performance, in this paper, we propose a unified
heterogeneous learning framework, which combines both weighted unsupervised
contrastive loss and weighted supervised contrastive loss to model multiple
types of heterogeneity. We also provide theoretical analyses showing that the
proposed weighted supervised contrastive loss is the lower bound of the mutual
information of two samples from the same class and the weighted unsupervised
contrastive loss is the lower bound of the mutual information between the
hidden representation of two views of the same sample. Experimental results on
real-world data sets demonstrate the effectiveness and the efficiency of the
proposed method modeling multiple types of heterogeneity.
Related papers
- Towards Multi-view Graph Anomaly Detection with Similarity-Guided Contrastive Clustering [35.1801853090859]
Anomaly detection on graphs plays an important role in many real-world applications.
We propose an autoencoder-based clustering framework regularized by a similarity-guided contrastive loss to detect anomalous nodes.
arXiv Detail & Related papers (2024-09-15T15:41:59Z) - Regularized Contrastive Partial Multi-view Outlier Detection [76.77036536484114]
We propose a novel method named Regularized Contrastive Partial Multi-view Outlier Detection (RCPMOD)
In this framework, we utilize contrastive learning to learn view-consistent information and distinguish outliers by the degree of consistency.
Experimental results on four benchmark datasets demonstrate that our proposed approach could outperform state-of-the-art competitors.
arXiv Detail & Related papers (2024-08-02T14:34:27Z) - Revealing Multimodal Contrastive Representation Learning through Latent
Partial Causal Models [85.67870425656368]
We introduce a unified causal model specifically designed for multimodal data.
We show that multimodal contrastive representation learning excels at identifying latent coupled variables.
Experiments demonstrate the robustness of our findings, even when the assumptions are violated.
arXiv Detail & Related papers (2024-02-09T07:18:06Z) - Hodge-Aware Contrastive Learning [101.56637264703058]
Simplicial complexes prove effective in modeling data with multiway dependencies.
We develop a contrastive self-supervised learning approach for processing simplicial data.
arXiv Detail & Related papers (2023-09-14T00:40:07Z) - Tackling Diverse Minorities in Imbalanced Classification [80.78227787608714]
Imbalanced datasets are commonly observed in various real-world applications, presenting significant challenges in training classifiers.
We propose generating synthetic samples iteratively by mixing data samples from both minority and majority classes.
We demonstrate the effectiveness of our proposed framework through extensive experiments conducted on seven publicly available benchmark datasets.
arXiv Detail & Related papers (2023-08-28T18:48:34Z) - Robust Direct Learning for Causal Data Fusion [14.462235940634969]
We provide a framework for integrating multi-source data that separates the treatment effect from other nuisance functions.
We also propose a causal information-aware weighting function motivated by theoretical insights from the semiparametric efficiency theory.
arXiv Detail & Related papers (2022-11-01T03:33:22Z) - Adversarial Lagrangian Integrated Contrastive Embedding for Limited Size
Datasets [8.926248371832852]
This study presents a novel adversarial Lagrangian integrated contrastive embedding (ALICE) method for small-sized datasets.
The accuracy improvement and training convergence of the proposed pre-trained adversarial transfer are shown.
A novel adversarial integrated contrastive model using various augmentation techniques is investigated.
arXiv Detail & Related papers (2022-10-06T23:59:28Z) - Variational Distillation for Multi-View Learning [104.17551354374821]
We design several variational information bottlenecks to exploit two key characteristics for multi-view representation learning.
Under rigorously theoretical guarantee, our approach enables IB to grasp the intrinsic correlation between observations and semantic labels.
arXiv Detail & Related papers (2022-06-20T03:09:46Z) - Scalable Regularised Joint Mixture Models [2.0686407686198263]
In many applications, data can be heterogeneous in the sense of spanning latent groups with different underlying distributions.
We propose an approach for heterogeneous data that allows joint learning of (i) explicit multivariate feature distributions, (ii) high-dimensional regression models and (iii) latent group labels.
The approach is demonstrably effective in high dimensions, combining data reduction for computational efficiency with a re-weighting scheme that retains key signals even when the number of features is large.
arXiv Detail & Related papers (2022-05-03T13:38:58Z) - Generalizable Information Theoretic Causal Representation [37.54158138447033]
We propose to learn causal representation from observational data by regularizing the learning procedure with mutual information measures according to our hypothetical causal graph.
The optimization involves a counterfactual loss, based on which we deduce a theoretical guarantee that the causality-inspired learning is with reduced sample complexity and better generalization ability.
arXiv Detail & Related papers (2022-02-17T00:38:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.