Propensity Score Alignment of Unpaired Multimodal Data

• URL: http://arxiv.org/abs/2404.01595v2
• Date: Tue, 29 Oct 2024 05:04:20 GMT
• Title: Propensity Score Alignment of Unpaired Multimodal Data
• Authors: Johnny Xi, Jana Osea, Zuheng Xu, Jason Hartford,
• Abstract summary: Multimodal representation learning techniques typically rely on paired samples to learn common representations. This paper presents an approach to address the challenge of aligning unpaired samples across disparate modalities in multimodal representation learning.
• Score: 3.8373578956681555
• License:
• Abstract: Multimodal representation learning techniques typically rely on paired samples to learn common representations, but paired samples are challenging to collect in fields such as biology where measurement devices often destroy the samples. This paper presents an approach to address the challenge of aligning unpaired samples across disparate modalities in multimodal representation learning. We draw an analogy between potential outcomes in causal inference and potential views in multimodal observations, which allows us to use Rubin's framework to estimate a common space in which to match samples. Our approach assumes we collect samples that are experimentally perturbed by treatments, and uses this to estimate a propensity score from each modality, which encapsulates all shared information between a latent state and treatment and can be used to define a distance between samples. We experiment with two alignment techniques that leverage this distance -- shared nearest neighbours (SNN) and optimal transport (OT) matching -- and find that OT matching results in significant improvements over state-of-the-art alignment approaches in both a synthetic multi-modal setting and in real-world data from NeurIPS Multimodal Single-Cell Integration Challenge.

Related papers

• Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning [20.491176017183044]
  This paper tackles the multi-objective reinforcement learning (MORL) problem. It introduces an innovative actor-critic algorithm named MOAC which finds a policy by iteratively making trade-offs among conflicting reward signals.
  arXiv  Detail & Related papers  (2024-05-05T23:52:57Z)
• Deep Generative Sampling in the Dual Divergence Space: A Data-efficient & Interpretative Approach for Generative AI [29.13807697733638]
  We build on the remarkable achievements in generative sampling of natural images. We propose an innovative challenge, potentially overly ambitious, which involves generating samples that resemble images. The statistical challenge lies in the small sample size, sometimes consisting of a few hundred subjects.
  arXiv  Detail & Related papers  (2024-04-10T22:35:06Z)
• Enhancing multimodal cooperation via sample-level modality valuation [10.677997431505815]
  We introduce a sample-level modality valuation metric to evaluate the contribution of each modality for each sample. Via modality valuation we observe that modality discrepancy indeed could be different at sample-level beyond the global contribution discrepancy at dataset-level. Our methods reasonably observe the fine-grained uni-modal contribution and achieve considerable improvement.
  arXiv  Detail & Related papers  (2023-09-12T14:16:34Z)
• Tackling Diverse Minorities in Imbalanced Classification [80.78227787608714]
  Imbalanced datasets are commonly observed in various real-world applications, presenting significant challenges in training classifiers. We propose generating synthetic samples iteratively by mixing data samples from both minority and majority classes. We demonstrate the effectiveness of our proposed framework through extensive experiments conducted on seven publicly available benchmark datasets.
  arXiv  Detail & Related papers  (2023-08-28T18:48:34Z)
• Detecting Adversarial Data by Probing Multiple Perturbations Using Expected Perturbation Score [62.54911162109439]
  Adversarial detection aims to determine whether a given sample is an adversarial one based on the discrepancy between natural and adversarial distributions. We propose a new statistic called expected perturbation score (EPS), which is essentially the expected score of a sample after various perturbations. We develop EPS-based maximum mean discrepancy (MMD) as a metric to measure the discrepancy between the test sample and natural samples.
  arXiv  Detail & Related papers  (2023-05-25T13:14:58Z)
• Revisiting the Evaluation of Image Synthesis with GANs [55.72247435112475]
  This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set.
  arXiv  Detail & Related papers  (2023-04-04T17:54:32Z)
• Deep Incomplete Multi-view Clustering with Cross-view Partial Sample and Prototype Alignment [50.82982601256481]
  We propose a Cross-view Partial Sample and Prototype Alignment Network (CPSPAN) for Deep Incomplete Multi-view Clustering. Unlike existing contrastive-based methods, we adopt pair-observed data alignment as 'proxy supervised signals' to guide instance-to-instance correspondence construction.
  arXiv  Detail & Related papers  (2023-03-28T02:31:57Z)
• Double-matched matrix decomposition for multi-view data [0.6091702876917281]
  We consider the problem of extracting joint and individual signals from multi-view data, that is data collected from different sources on matched samples. Our proposed double-matched matrix decomposition allows to simultaneously extract joint and individual signals across subjects. We apply our method to data from the English Premier League soccer matches, and find joint and individual multi-view signals that align with domain specific knowledge.
  arXiv  Detail & Related papers  (2021-05-07T17:09:57Z)
• Jo-SRC: A Contrastive Approach for Combating Noisy Labels [58.867237220886885]
  We propose a noise-robust approach named Jo-SRC (Joint Sample Selection and Model Regularization based on Consistency) Specifically, we train the network in a contrastive learning manner. Predictions from two different views of each sample are used to estimate its "likelihood" of being clean or out-of-distribution.
  arXiv  Detail & Related papers  (2021-03-24T07:26:07Z)
• Hierarchical Optimal Transport for Robust Multi-View Learning [97.21355697826345]
  Two assumptions may be questionable in practice, which limits the application of multi-view learning. We propose a hierarchical optimal transport (HOT) method to mitigate the dependency on these two assumptions. The HOT method is applicable to both unsupervised and semi-supervised learning, and experimental results show that it performs robustly on both synthetic and real-world tasks.
  arXiv  Detail & Related papers  (2020-06-04T22:24:45Z)
• M$^5$L: Multi-Modal Multi-Margin Metric Learning for RGBT Tracking [44.296318907168]
  Classifying the confusing samples in the course of RGBT tracking is a challenging problem. We propose a novel Multi-Modal Multi-Margin Metric Learning framework, named M$5$L for RGBT tracking. Our framework clearly improves the tracking performance and outperforms the state-of-the-art RGBT trackers.
  arXiv  Detail & Related papers  (2020-03-17T11:37:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.