Learning Shared Representations from Unpaired Data

• URL: http://arxiv.org/abs/2505.21524v1
• Date: Fri, 23 May 2025 11:13:04 GMT
• Title: Learning Shared Representations from Unpaired Data
• Authors: Amitai Yacobi, Nir Ben-Ari, Ronen Talmon, Uri Shaham,
• Abstract summary: We show that shared representations can be learned almost exclusively from unpaired data.<n> Empirical results in computer vision and natural language processing domains support its potential.
• Score: 8.370305493567542
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning shared representations is a primary area of multimodal representation learning. The current approaches to achieve a shared embedding space rely heavily on paired samples from each modality, which are significantly harder to obtain than unpaired ones. In this work, we demonstrate that shared representations can be learned almost exclusively from unpaired data. Our arguments are grounded in the spectral embeddings of the random walk matrices constructed independently from each unimodal representation. Empirical results in computer vision and natural language processing domains support its potential, revealing the effectiveness of unpaired data in capturing meaningful cross-modal relations, demonstrating high capabilities in retrieval tasks, generation, arithmetics, zero-shot, and cross-domain classification. This work, to the best of our knowledge, is the first to demonstrate these capabilities almost exclusively from unpaired samples, giving rise to a cross-modal embedding that could be viewed as universal, i.e., independent of the specific modalities of the data. Our code IS publicly available at https://github.com/shaham-lab/SUE.

Related papers

• Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities [21.745266161874195]
  Contrastive learning methods, such as CLIP, leverage naturally paired data-for example, images and their corresponding text captions-to learn general representations that transfer efficiently to downstream tasks. We show that the pairwise application of CLIP fails to capture joint information between modalities, thereby limiting the quality of the learned representations. We present Symile, a simple contrastive learning approach that captures higher-order information between any number of modalities.
  arXiv  Detail & Related papers  (2024-11-01T21:49:25Z)
• Cross-Modal Few-Shot Learning: a Generative Transfer Learning Framework [58.362064122489166]
  This paper introduces the Cross-modal Few-Shot Learning task, which aims to recognize instances across multiple modalities while relying on scarce labeled data.<n>We propose a Generative Transfer Learning framework by simulating how humans abstract and generalize concepts.<n>We show that the GTL achieves state-of-the-art performance across seven multi-modal datasets across RGB-Sketch, RGB-Infrared, and RGB-Depth.
  arXiv  Detail & Related papers  (2024-10-14T16:09:38Z)
• Collaborative Learning with Different Labeling Functions [7.228285747845779]
  We study a variant of Collaborative PAC Learning, in which we aim to learn an accurate classifier for each of the $n$ data distributions. We show that, when the data distributions satisfy a weaker realizability assumption, sample-efficient learning is still feasible.
  arXiv  Detail & Related papers  (2024-02-16T04:32:22Z)
• Semi-supervised Multimodal Representation Learning through a Global Workspace [2.8948274245812335]
  "Global Workspace" is a shared representation for two input modalities. This architecture is amenable to self-supervised training via cycle-consistency. We show that such an architecture can be trained to align and translate between two modalities with very little need for matched data.
  arXiv  Detail & Related papers  (2023-06-27T12:41:36Z)
• Leveraging sparse and shared feature activations for disentangled representation learning [112.22699167017471]
  We propose to leverage knowledge extracted from a diversified set of supervised tasks to learn a common disentangled representation. We validate our approach on six real world distribution shift benchmarks, and different data modalities.
  arXiv  Detail & Related papers  (2023-04-17T01:33:24Z)
• Non-contrastive representation learning for intervals from well logs [58.70164460091879]
  The representation learning problem in the oil & gas industry aims to construct a model that provides a representation based on logging data for a well interval. One of the possible approaches is self-supervised learning (SSL) We are the first to introduce non-contrastive SSL for well-logging data.
  arXiv  Detail & Related papers  (2022-09-28T13:27:10Z)
• Does a Technique for Building Multimodal Representation Matter? -- Comparative Analysis [0.0]
  We show that the choice of the technique for building multimodal representation is crucial to obtain the highest possible model's performance. Experiments are conducted on three datasets: Amazon Reviews, MovieLens25M, and MovieLens1M.
  arXiv  Detail & Related papers  (2022-06-09T21:30:10Z)
• Exploiting Shared Representations for Personalized Federated Learning [54.65133770989836]
  We propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client. Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation. This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions.
  arXiv  Detail & Related papers  (2021-02-14T05:36:25Z)
• Relation-Guided Representation Learning [53.60351496449232]
  We propose a new representation learning method that explicitly models and leverages sample relations. Our framework well preserves the relations between samples. By seeking to embed samples into subspace, we show that our method can address the large-scale and out-of-sample problem.
  arXiv  Detail & Related papers  (2020-07-11T10:57:45Z)
• Evolving Losses for Unsupervised Video Representation Learning [91.2683362199263]
  We present a new method to learn video representations from large-scale unlabeled video data. The proposed unsupervised representation learning results in a single RGB network and outperforms previous methods.
  arXiv  Detail & Related papers  (2020-02-26T16:56:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.