Balance-aware Sequence Sampling Makes Multi-modal Learning Better

• URL: http://arxiv.org/abs/2501.01470v1
• Date: Wed, 01 Jan 2025 06:19:55 GMT
• Title: Balance-aware Sequence Sampling Makes Multi-modal Learning Better
• Authors: Zhi-Hao Guan,
• Abstract summary: We propose Balance-aware Sequence Sampling (BSS) to enhance the robustness of MML.<n>Via a multi-perspective measurer, we first define a multi-perspective measurer to evaluate the balance degree of each sample.<n>We employ a scheduler based on curriculum learning (CL) that incrementally provides training subsets, progressing from balanced to imbalanced samples to rebalance MML.
• Score: 0.5439020425819
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To address the modality imbalance caused by data heterogeneity, existing multi-modal learning (MML) approaches primarily focus on balancing this difference from the perspective of optimization objectives. However, almost all existing methods ignore the impact of sample sequences, i.e., an inappropriate training order tends to trigger learning bias in the model, further exacerbating modality imbalance. In this paper, we propose Balance-aware Sequence Sampling (BSS) to enhance the robustness of MML. Specifically, we first define a multi-perspective measurer to evaluate the balance degree of each sample. Via the evaluation, we employ a heuristic scheduler based on curriculum learning (CL) that incrementally provides training subsets, progressing from balanced to imbalanced samples to rebalance MML. Moreover, considering that sample balance may evolve as the model capability increases, we propose a learning-based probabilistic sampling method to dynamically update the training sequences at the epoch level, further improving MML performance. Extensive experiments on widely used datasets demonstrate the superiority of our method compared with state-of-the-art (SOTA) MML approaches.

Related papers

• Injecting Imbalance Sensitivity for Multi-Task Learning [36.60453299563175]
  Multi-task learning (MTL) has emerged as a promising approach for deploying deep learning models in real-life applications. Recent studies have proposed optimization-based learning paradigms to establish task-shared representations in MTL. Our paper empirically argues that these studies primarily emphasize the conflict issue while neglecting the potentially more significant impact of imbalance/dominance in MTL.
  arXiv  Detail & Related papers  (2025-03-11T03:11:54Z)
• DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learning [15.524342129628957]
  DynCIM is a novel dynamic curriculum learning framework designed to quantify the inherent imbalances from both sample and modality perspectives. DynCIM employs a sample-level curriculum to dynamically assess each sample's difficulty according to prediction deviation, consistency, and stability. A modality-level curriculum measures modality contributions from global and local.
  arXiv  Detail & Related papers  (2025-03-09T05:30:15Z)
• Rebalanced Multimodal Learning with Data-aware Unimodal Sampling [39.77348232514481]
  We propose a novel MML approach called underlineData-aware underlineUnimodal underlineSampling(method) Based on the learning status, we propose a reinforcement learning(RL)-based data-aware unimodal sampling approaches. Our method can be seamlessly incorporated into almost all existing multimodal learning approaches as a plugin.
  arXiv  Detail & Related papers  (2025-03-05T08:19:31Z)
• Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion [6.621745547882088]
  The existence of modality imbalance hinders multimodal learning from achieving its expected superiority over unimodal models in practice. By designing a sustained boosting algorithm, we propose a novel multimodal learning approach to balance the classification ability of weak and strong modalities.
  arXiv  Detail & Related papers  (2025-02-27T14:12:20Z)
• Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining [55.262510814326035]
  Existing reweighting strategies primarily focus on group-level data importance. We introduce novel algorithms for dynamic, instance-level data reweighting. Our framework allows us to devise reweighting strategies deprioritizing redundant or uninformative data.
  arXiv  Detail & Related papers  (2025-02-10T17:57:15Z)
• SeMi: When Imbalanced Semi-Supervised Learning Meets Mining Hard Examples [54.760757107700755]
  Semi-Supervised Learning (SSL) can leverage abundant unlabeled data to boost model performance. The class-imbalanced data distribution in real-world scenarios poses great challenges to SSL, resulting in performance degradation. We propose a method that enhances the performance of Imbalanced Semi-Supervised Learning by Mining Hard Examples (SeMi)
  arXiv  Detail & Related papers  (2025-01-10T14:35:16Z)
• On-the-fly Modulation for Balanced Multimodal Learning [53.616094855778954]
  Multimodal learning is expected to boost model performance by integrating information from different modalities. The widely-used joint training strategy leads to imbalanced and under-optimized uni-modal representations. We propose On-the-fly Prediction Modulation (OPM) and On-the-fly Gradient Modulation (OGM) strategies to modulate the optimization of each modality.
  arXiv  Detail & Related papers  (2024-10-15T13:15:50Z)
• Multimodal Classification via Modal-Aware Interactive Enhancement [6.621745547882088]
  We propose a novel multimodal learning method, called modal-aware interactive enhancement (MIE) Specifically, we first utilize an optimization strategy based on sharpness aware minimization (SAM) to smooth the learning objective during the forward phase. Then, with the help of the geometry property of SAM, we propose a gradient modification strategy to impose the influence between different modalities during the backward phase.
  arXiv  Detail & Related papers  (2024-07-05T15:32:07Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• BEM: Balanced and Entropy-based Mix for Long-Tailed Semi-Supervised Learning [21.53320689054414]
  This paper introduces the Balanced and Entropy-based Mix (BEM), a pioneering mixing approach to re-balance the class distribution of both data quantity and uncertainty. Experimental results show that BEM significantly enhances various LTSSL frameworks and achieves state-of-the-art performances across multiple benchmarks.
  arXiv  Detail & Related papers  (2024-04-01T15:31:04Z)
• Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization [165.98557106089777]
  A key challenge is to enhance the capabilities of large language models (LLMs) amid a looming shortage of high-quality training data. Our study starts from an empirical strategy for the light continual training of LLMs using their original pre-training data sets. We then formalize this strategy into a principled framework of Instance-Reweighted Distributionally Robust Optimization.
  arXiv  Detail & Related papers  (2024-02-22T04:10:57Z)
• Mixed Semi-Supervised Generalized-Linear-Regression with Applications to Deep-Learning and Interpolators [6.537685198688539]
  We present a methodology for using unlabeled data to design semi supervised learning (SSL) methods.<n>We include in each of them a mixing parameter $alpha$, controlling the weight given to the unlabeled data.<n>We demonstrate the effectiveness of our methodology in delivering substantial improvement compared to the standard supervised models.
  arXiv  Detail & Related papers  (2023-02-19T09:55:18Z)
• Learning to Re-weight Examples with Optimal Transport for Imbalanced Classification [74.62203971625173]
  Imbalanced data pose challenges for deep learning based classification models. One of the most widely-used approaches for tackling imbalanced data is re-weighting. We propose a novel re-weighting method based on optimal transport (OT) from a distributional point of view.
  arXiv  Detail & Related papers  (2022-08-05T01:23:54Z)
• Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation [93.52573037053449]
  H-MARL (Hallucinated Multi-Agent Reinforcement Learning) learns successful equilibrium policies after a few interactions with the environment. We demonstrate our approach experimentally on an autonomous driving simulation benchmark.
  arXiv  Detail & Related papers  (2022-03-14T17:24:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.