Active Multimodal Distillation for Few-shot Action Recognition

• URL: http://arxiv.org/abs/2506.13322v1
• Date: Mon, 16 Jun 2025 10:10:56 GMT
• Title: Active Multimodal Distillation for Few-shot Action Recognition
• Authors: Weijia Feng, Yichen Zhu, Ruojia Zhang, Chenyang Wang, Fei Ma, Xiaobao Wang, Xiaobai Li,
• Abstract summary: This paper presents a novel framework that actively identifies reliable modalities for each sample using task-specific contextual cues.<n>Our framework integrates an Active Sample Inference (ASI) module, which utilizes active inference to predict reliable modalities.<n>Unlike reinforcement learning, active inference replaces rewards with evidence-based preferences, making more stable predictions.
• Score: 19.872938560809988
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Owing to its rapid progress and broad application prospects, few-shot action recognition has attracted considerable interest. However, current methods are predominantly based on limited single-modal data, which does not fully exploit the potential of multimodal information. This paper presents a novel framework that actively identifies reliable modalities for each sample using task-specific contextual cues, thus significantly improving recognition performance. Our framework integrates an Active Sample Inference (ASI) module, which utilizes active inference to predict reliable modalities based on posterior distributions and subsequently organizes them accordingly. Unlike reinforcement learning, active inference replaces rewards with evidence-based preferences, making more stable predictions. Additionally, we introduce an active mutual distillation module that enhances the representation learning of less reliable modalities by transferring knowledge from more reliable ones. Adaptive multimodal inference is employed during the meta-test to assign higher weights to reliable modalities. Extensive experiments across multiple benchmarks demonstrate that our method significantly outperforms existing approaches.

Related papers

• Asymmetric Reinforcing against Multi-modal Representation Bias [59.685072206359855]
  We propose an Asymmetric Reinforcing method against Multimodal representation bias (ARM)<n>Our ARM dynamically reinforces the weak modalities while maintaining the ability to represent dominant modalities through conditional mutual information.<n>We have significantly improved the performance of multimodal learning, making notable progress in mitigating imbalanced multimodal learning.
  arXiv  Detail & Related papers  (2025-01-02T13:00:06Z)
• Beyond Unimodal Learning: The Importance of Integrating Multiple Modalities for Lifelong Learning [23.035725779568587]
  We study the role and interactions of multiple modalities in mitigating forgetting in deep neural networks (DNNs) Our findings demonstrate that leveraging multiple views and complementary information from multiple modalities enables the model to learn more accurate and robust representations. We propose a method for integrating and aligning the information from different modalities by utilizing the relational structural similarities between the data points in each modality.
  arXiv  Detail & Related papers  (2024-05-04T22:02:58Z)
• Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding [62.70450216120704]
  Unsupervised pre-training has shown great success in skeleton-based action understanding. We propose a Unified Multimodal Unsupervised Representation Learning framework, called UmURL. UmURL exploits an efficient early-fusion strategy to jointly encode the multi-modal features in a single-stream manner.
  arXiv  Detail & Related papers  (2023-11-06T13:56:57Z)
• Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Tasks [92.32670915472099]
  We propose an ensemble diversification framework exploiting the generation of synthetic counterfactuals using Diffusion Probabilistic Models (DPMs) We show that diffusion-guided diversification can lead models to avert attention from shortcut cues, achieving ensemble diversity performance comparable to previous methods requiring additional data collection.
  arXiv  Detail & Related papers  (2023-10-03T17:37:52Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• Elevating Skeleton-Based Action Recognition with Efficient Multi-Modality Self-Supervision [40.16465314639641]
  Self-supervised representation learning for human action recognition has developed rapidly in recent years. Most of the existing works are based on skeleton data while using a multi-modality setup. We first propose an Implicit Knowledge Exchange Module which alleviates the propagation of erroneous knowledge between low-performance modalities.
  arXiv  Detail & Related papers  (2023-09-21T12:27:43Z)
• Mimicking Better by Matching the Approximate Action Distribution [48.95048003354255]
  We introduce MAAD, a novel, sample-efficient on-policy algorithm for Imitation Learning from Observations. We show that it requires considerable fewer interactions to achieve expert performance, outperforming current state-of-the-art on-policy methods.
  arXiv  Detail & Related papers  (2023-06-16T12:43:47Z)
• Calibrating Multimodal Learning [94.65232214643436]
  We propose a novel regularization technique, i.e., Calibrating Multimodal Learning (CML) regularization, to calibrate the predictive confidence of previous methods. This technique could be flexibly equipped by existing models and improve the performance in terms of confidence calibration, classification accuracy, and model robustness.
  arXiv  Detail & Related papers  (2023-06-02T04:29:57Z)
• Cross-modal Contrastive Learning for Multimodal Fake News Detection [10.760000041969139]
  COOLANT is a cross-modal contrastive learning framework for multimodal fake news detection. A cross-modal fusion module is developed to learn the cross-modality correlations. An attention guidance module is implemented to help effectively and interpretably aggregate the aligned unimodal representations.
  arXiv  Detail & Related papers  (2023-02-25T10:12:34Z)
• Active Speaker Detection as a Multi-Objective Optimization with Uncertainty-based Multimodal Fusion [0.07874708385247352]
  This paper outlines active speaker detection as a multi-objective learning problem to leverage best of each modalities using a novel self-attention, uncertainty-based multimodal fusion scheme. Results obtained show that the proposed multi-objective learning architecture outperforms traditional approaches in improving both mAP and AUC scores.
  arXiv  Detail & Related papers  (2021-06-07T17:38:55Z)
• Robust Latent Representations via Cross-Modal Translation and Alignment [36.67937514793215]
  Most multi-modal machine learning methods require that all the modalities used for training are also available for testing. To address this limitation, we aim to improve the testing performance of uni-modal systems using multiple modalities during training only. The proposed multi-modal training framework uses cross-modal translation and correlation-based latent space alignment.
  arXiv  Detail & Related papers  (2020-11-03T11:18:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.