Towards Adaptive Fusion of Multimodal Deep Networks for Human Action Recognition

• URL: http://arxiv.org/abs/2512.04943v1
• Date: Thu, 04 Dec 2025 16:09:45 GMT
• Title: Towards Adaptive Fusion of Multimodal Deep Networks for Human Action Recognition
• Authors: Novanto Yudistira,
• Abstract summary: This study introduces a pioneering methodology for human action recognition by harnessing deep neural network techniques and adaptive fusion strategies.<n>Gating mechanisms facilitate the extraction of pivotal features, resulting in a more holistic representation of actions.<n>The significance of this research lies in its potential to revolutionize action recognition systems across diverse fields.
• Score: 3.756550107432323
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This study introduces a pioneering methodology for human action recognition by harnessing deep neural network techniques and adaptive fusion strategies across multiple modalities, including RGB, optical flows, audio, and depth information. Employing gating mechanisms for multimodal fusion, we aim to surpass limitations inherent in traditional unimodal recognition methods while exploring novel possibilities for diverse applications. Through an exhaustive investigation of gating mechanisms and adaptive weighting-based fusion architectures, our methodology enables the selective integration of relevant information from various modalities, thereby bolstering both accuracy and robustness in action recognition tasks. We meticulously examine various gated fusion strategies to pinpoint the most effective approach for multimodal action recognition, showcasing its superiority over conventional unimodal methods. Gating mechanisms facilitate the extraction of pivotal features, resulting in a more holistic representation of actions and substantial enhancements in recognition performance. Our evaluations across human action recognition, violence action detection, and multiple self-supervised learning tasks on benchmark datasets demonstrate promising advancements in accuracy. The significance of this research lies in its potential to revolutionize action recognition systems across diverse fields. The fusion of multimodal information promises sophisticated applications in surveillance and human-computer interaction, especially in contexts related to active assisted living.

Related papers

• Incorporating brain-inspired mechanisms for multimodal learning in artificial intelligence [12.09002670544188]
  The brain exhibits an inverse effectiveness phenomenon, wherein weaker unimodal cues yield stronger multisensory integration benefits.<n>Inspired by this biological mechanism, we propose an inverse effectiveness driven multimodal fusion (IEMF) strategy.<n>By incorporating this strategy into neural networks, we achieve more efficient integration with improved model performance and computational efficiency.
  arXiv  Detail & Related papers  (2025-05-15T11:08:50Z)
• Process Optimization and Deployment for Sensor-Based Human Activity Recognition Based on Deep Learning [9.445469731895505]
  We propose a comprehensive optimization process approach centered on multi-attention interaction.<n>We conduct extensive testing on three public datasets, including ablation studies, comparisons of related work and embedded deployments.
  arXiv  Detail & Related papers  (2025-03-22T16:48:16Z)
• A Novel Approach to for Multimodal Emotion Recognition : Multimodal semantic information fusion [3.1409950035735914]
  This paper proposes a novel multimodal emotion recognition approach, DeepMSI-MER, based on the integration of contrastive learning and visual sequence compression.<n> Experimental results on two public datasets, IEMOCAP and MELD, demonstrate that DeepMSI-MER significantly improves the accuracy and robustness of emotion recognition.
  arXiv  Detail & Related papers  (2025-02-12T17:07:43Z)
• Progressive Multimodal Reasoning via Active Retrieval [64.74746997923967]
  Multi-step multimodal reasoning tasks pose significant challenges for large language models (MLLMs)<n>We propose AR-MCTS, a universal framework designed to progressively improve the reasoning capabilities of MLLMs.<n>We show that AR-MCTS can optimize sampling diversity and accuracy, yielding reliable multimodal reasoning.
  arXiv  Detail & Related papers  (2024-12-19T13:25:39Z)
• Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition [52.522244807811894]
  We propose a novel multimodal Transformer framework using prompt learning to address the issue of missing modalities. Our method introduces three types of prompts: generative prompts, missing-signal prompts, and missing-type prompts. Through prompt learning, we achieve a substantial reduction in the number of trainable parameters.
  arXiv  Detail & Related papers  (2024-07-07T13:55:56Z)
• Joint Multimodal Transformer for Emotion Recognition in the Wild [49.735299182004404]
  Multimodal emotion recognition (MMER) systems typically outperform unimodal systems. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention.
  arXiv  Detail & Related papers  (2024-03-15T17:23:38Z)
• Egocentric RGB+Depth Action Recognition in Industry-Like Settings [50.38638300332429]
  Our work focuses on recognizing actions from egocentric RGB and Depth modalities in an industry-like environment. Our framework is based on the 3D Video SWIN Transformer to encode both RGB and Depth modalities effectively. Our method also secured first place at the multimodal action recognition challenge at ICIAP 2023.
  arXiv  Detail & Related papers  (2023-09-25T08:56:22Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection [57.13665112065285]
  Human-Object Interaction (HOI) detection is a challenging computer vision task. We present a framework that enhances HOI detection by incorporating structured text knowledge.
  arXiv  Detail & Related papers  (2023-07-25T14:20:52Z)
• Alternative Telescopic Displacement: An Efficient Multimodal Alignment Method [3.0903319879656084]
  This paper introduces an innovative approach to feature alignment that revolutionizes the fusion of multimodal information. Our method employs a novel iterative process of telescopic displacement and expansion of feature representations across different modalities, culminating in a coherent unified representation within a shared feature space.
  arXiv  Detail & Related papers  (2023-06-29T13:49:06Z)
• Recent Progress in Appearance-based Action Recognition [73.6405863243707]
  Action recognition is a task to identify various human actions in a video. Recent appearance-based methods have achieved promising progress towards accurate action recognition.
  arXiv  Detail & Related papers  (2020-11-25T10:18:12Z)
• Deep Auto-Encoders with Sequential Learning for Multimodal Dimensional Emotion Recognition [38.350188118975616]
  We propose a novel deep neural network architecture consisting of a two-stream auto-encoder and a long short term memory for emotion recognition. We carry out extensive experiments on the multimodal emotion in the wild dataset: RECOLA. Experimental results show that the proposed method achieves state-of-the-art recognition performance and surpasses existing schemes by a significant margin.
  arXiv  Detail & Related papers  (2020-04-28T01:25:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.