Greedy Modality Selection via Approximate Submodular Maximization

• URL: http://arxiv.org/abs/2210.12562v1
• Date: Sat, 22 Oct 2022 22:07:27 GMT
• Title: Greedy Modality Selection via Approximate Submodular Maximization
• Authors: Runxiang Cheng, Gargi Balasubramaniam, Yifei He, Yao-Hung Hubert Tsai, Han Zhao
• Abstract summary: Multimodal learning considers learning from multi-modality data, aiming to fuse heterogeneous sources of information. It is not always feasible to leverage all available modalities due to memory constraints. We study modality selection, intending to efficiently select the most informative and complementary modalities under certain computational constraints.
• Score: 19.22947539760366
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal learning considers learning from multi-modality data, aiming to fuse heterogeneous sources of information. However, it is not always feasible to leverage all available modalities due to memory constraints. Further, training on all the modalities may be inefficient when redundant information exists within data, such as different subsets of modalities providing similar performance. In light of these challenges, we study modality selection, intending to efficiently select the most informative and complementary modalities under certain computational constraints. We formulate a theoretical framework for optimizing modality selection in multimodal learning and introduce a utility measure to quantify the benefit of selecting a modality. For this optimization problem, we present efficient algorithms when the utility measure exhibits monotonicity and approximate submodularity. We also connect the utility measure with existing Shapley-value-based feature importance scores. Last, we demonstrate the efficacy of our algorithm on synthetic (Patch-MNIST) and two real-world (PEMS-SF, CMU-MOSI) datasets.

Related papers

• On-the-fly Modulation for Balanced Multimodal Learning [53.616094855778954]
  Multimodal learning is expected to boost model performance by integrating information from different modalities. The widely-used joint training strategy leads to imbalanced and under-optimized uni-modal representations. We propose On-the-fly Prediction Modulation (OPM) and On-the-fly Gradient Modulation (OGM) strategies to modulate the optimization of each modality.
  arXiv  Detail & Related papers  (2024-10-15T13:15:50Z)
• Prioritizing Modalities: Flexible Importance Scheduling in Federated Multimodal Learning [5.421492821020181]
  Federated Learning (FL) is a distributed machine learning approach that enables devices to collaboratively train models without sharing their local data. Applying FL to real-world data presents challenges, particularly as most existing FL research focuses on unimodal data. We propose FlexMod, a novel approach to enhance computational efficiency in MFL by adaptively allocating training resources for each modality encoder.
  arXiv  Detail & Related papers  (2024-08-13T01:14:27Z)
• Enhancing Neural Subset Selection: Integrating Background Information into Set Representations [53.15923939406772]
  We show that when the target value is conditioned on both the input set and subset, it is essential to incorporate an textitinvariant sufficient statistic of the superset into the subset of interest. This ensures that the output value remains invariant to permutations of the subset and its corresponding superset, enabling identification of the specific superset from which the subset originated.
  arXiv  Detail & Related papers  (2024-02-05T16:09:35Z)
• Amortizing intractable inference in large language models [56.92471123778389]
  We use amortized Bayesian inference to sample from intractable posterior distributions. We empirically demonstrate that this distribution-matching paradigm of LLM fine-tuning can serve as an effective alternative to maximum-likelihood training. As an important application, we interpret chain-of-thought reasoning as a latent variable modeling problem.
  arXiv  Detail & Related papers  (2023-10-06T16:36:08Z)
• Multi-Resolution Active Learning of Fourier Neural Operators [33.63483360957646]
  We propose Multi-Resolution Active learning of FNO (MRA-FNO), which can dynamically select the input functions and resolutions to lower the data cost as much as possible. Specifically, we propose a probabilistic multi-resolution FNO and use ensemble Monte-Carlo to develop an effective posterior inference algorithm. We have shown the advantage of our method in several benchmark operator learning tasks.
  arXiv  Detail & Related papers  (2023-09-29T04:41:27Z)
• Learning Unseen Modality Interaction [54.23533023883659]
  Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved.
  arXiv  Detail & Related papers  (2023-06-22T10:53:10Z)
• Learning to Maximize Mutual Information for Dynamic Feature Selection [13.821253491768168]
  We consider the dynamic feature selection (DFS) problem where a model sequentially queries features based on the presently available information. We explore a simpler approach of greedily selecting features based on their conditional mutual information. The proposed method is shown to recover the greedy policy when trained to optimality, and it outperforms numerous existing feature selection methods in our experiments.
  arXiv  Detail & Related papers  (2023-01-02T08:31:56Z)
• HyperImpute: Generalized Iterative Imputation with Automatic Model Selection [77.86861638371926]
  We propose a generalized iterative imputation framework for adaptively and automatically configuring column-wise models. We provide a concrete implementation with out-of-the-box learners, simulators, and interfaces.
  arXiv  Detail & Related papers  (2022-06-15T19:10:35Z)
• Maximum Likelihood Estimation for Multimodal Learning with Missing Modality [10.91899856969822]
  We propose an efficient approach based on maximum likelihood estimation to incorporate the knowledge in the modality-missing data. Our results demonstrate the effectiveness of the proposed approach, even when 95% of the training data has missing modality.
  arXiv  Detail & Related papers  (2021-08-24T03:50:54Z)
• Provable Multi-Objective Reinforcement Learning with Generative Models [98.19879408649848]
  We study the problem of single policy MORL, which learns an optimal policy given the preference of objectives. Existing methods require strong assumptions such as exact knowledge of the multi-objective decision process. We propose a new algorithm called model-based envelop value (EVI) which generalizes the enveloped multi-objective $Q$-learning algorithm.
  arXiv  Detail & Related papers  (2020-11-19T22:35:31Z)
• Adaptive Matching of Kernel Means [10.377914682543903]
  kernel mean matching (KMM) has become an important method for knowledge discovery and novelty detection in kernel machines. In this work, a novel approach to adaptive matching of kernel means is proposed, and selected data with high importance are adopted to achieve calculation efficiency with optimization. The experimental results on a wide variety of real-world data sets demonstrate the proposed method is able to give outstanding performance compared with several state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-11-16T09:00:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.