MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learning

• URL: http://arxiv.org/abs/2606.22220v2
• Date: Tue, 23 Jun 2026 14:03:33 GMT
• Title: MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learning
• Abstract summary: Memorization in machine learning models enables high performance on rare in-distribution samples.<n>It also causes harmful retention of noise and outliers, degrading generalization.<n>We introduce MultiMem, the first metric designed to quantify memorization in multi-modal contrastive learning.
• Score: 42.60499995994591
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Memorization in machine learning models enables high performance on rare in-distribution samples by capturing their atypical patterns. However, it also causes harmful retention of noise and outliers, degrading generalization. While memorization has been extensively studied in both supervised and self-supervised learning in the vision domain, it remains unexplored in multi-modal contrastive learning. We address this gap by introducing MultiMem, the first metric designed to quantify memorization in multi-modal contrastive learning. Through our systematic analysis, we demonstrate that cross-modal semantic misalignment has the strongest influence on memorization, with text being the dominant modality driving memorization, followed by video, image, and audio. We show that targeted augmentations applied across all modalities effectively reduce memorization as measured by our MultiMem metric and improve model performance. Overall, this work establishes the first framework for measuring and mitigating memorization in multi-modal contrastive learning, preventing harmful data retention and contributing to higher-performing models.

Related papers

• Understanding the Role of Rehearsal Scale in Continual Learning under Varying Model Capacities [11.882528379148141]
  We formulate rehearsal-based continual learning as a multidimensional effectiveness-driven iterative optimization problem.<n>We derive a closed-form analysis of adaptability, memorability, and generalization from the perspective of rehearsal scale.<n>We validate these insights through numerical simulations and extended analyses on deep neural networks across multiple real-world datasets.
  arXiv  Detail & Related papers  (2026-02-24T11:29:12Z)
• Human-Inspired Learning for Large Language Models via Obvious Record and Maximum-Entropy Method Discovery [0.11844977816228043]
  This paper proposes a human-inspired learning framework that integrates two complementary mechanisms.<n>The first, Obvious Record, explicitly stores cause--result (or question--solution) relationships as symbolic memory.<n>The second, Maximum-Entropy Method Discovery, prioritizes and preserves methods with high semantic dissimilarity.
  arXiv  Detail & Related papers  (2025-12-14T09:12:09Z)
• Quantifying Cross-Modality Memorization in Vision-Language Models [86.82366725590508]
  We study the unique characteristics of cross-modality memorization and conduct a systematic study centered on vision-language models.<n>Our results reveal that facts learned in one modality transfer to the other, but a significant gap exists between recalling information in the source and target modalities.
  arXiv  Detail & Related papers  (2025-06-05T16:10:47Z)
• PAL: Prompting Analytic Learning with Missing Modality for Multi-Modal Class-Incremental Learning [42.00851701431368]
  Multi-modal class-incremental learning (MMCIL) seeks to leverage multi-modal data, such as audio-visual and image-text pairs.<n>A critical challenge remains: the issue of missing modalities during incremental learning phases.<n>We propose PAL, a novel exemplar-free framework tailored to MMCIL under missing-modality scenarios.
  arXiv  Detail & Related papers  (2025-01-16T08:04:04Z)
• Exploring Memorization in Fine-tuned Language Models [53.52403444655213]
  We conduct the first comprehensive analysis to explore language models' memorization during fine-tuning across tasks. Our studies with open-sourced and our own fine-tuned LMs across various tasks indicate that memorization presents a strong disparity among different fine-tuning tasks. We provide an intuitive explanation of this task disparity via sparse coding theory and unveil a strong correlation between memorization and attention score distribution.
  arXiv  Detail & Related papers  (2023-10-10T15:41:26Z)
• Learning Unseen Modality Interaction [54.23533023883659]
  Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved.
  arXiv  Detail & Related papers  (2023-06-22T10:53:10Z)
• Learning Transferable Adversarial Robust Representations via Multi-view Consistency [57.73073964318167]
  We propose a novel meta-adversarial multi-view representation learning framework with dual encoders. We demonstrate the effectiveness of our framework on few-shot learning tasks from unseen domains.
  arXiv  Detail & Related papers  (2022-10-19T11:48:01Z)
• On Modality Bias Recognition and Reduction [70.69194431713825]
  We study the modality bias problem in the context of multi-modal classification. We propose a plug-and-play loss function method, whereby the feature space for each label is adaptively learned. Our method yields remarkable performance improvements compared with the baselines.
  arXiv  Detail & Related papers  (2022-02-25T13:47:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.