Deep Multimodal Fusion Detection through Spatial Mask and Channel Fusion

• URL: http://arxiv.org/abs/2608.02092v1
• Date: Mon, 03 Aug 2026 11:51:12 GMT
• Title: Deep Multimodal Fusion Detection through Spatial Mask and Channel Fusion
• Abstract summary: This paper proposes an Attention-Driven Complementarity Resampling framework for robust improvement of cross-modality object detection.<n>Based on a shared channel spatial attention mechanism, we first introduce the semantic mask exchange to actively mix the boundaries of the modalities.<n>Then we propose a learnable channel competition to sample and aggregate features in a channel-wise and learnable way.
• Score: 4.909774323398836
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Deep multimodal fusion for object detection has demonstrated good performance through mining modal characteristics. However, existing feature-level fusion methods mainly weigh between two modalities and unify them in a unified representation space. This can lead to overfitting or over-specialization of the statistical properties of a single modality within a dual-backbone architecture. This paper proposes an Attention-Driven Complementarity Resampling framework for robust improvement of cross-modality object detection. Based on a shared channel spatial attention mechanism, we first introduce the semantic mask exchange to actively mix the boundaries of the modalities during the training phase, forcing the backbone network to learn generalized features without relying on fixed modal labels. Then we propose a learnable channel competition to sample and aggregate features in a channel-wise and learnable way. Our experiments on multiple datasets demonstrate that the proposed method is effective and yields competitive results among existing state-of-the-art approaches. The source code is provided in the supplementary material.

Related papers

• DMFNet: Dual-Backbone Multiscale Fusion Network for Urban Scene Classification [1.529342790344802]
  DMFNet is a dual-backbone multiscale feature fusion framework with residual feature propagation and spatial attention for remote sensing scene classification.<n> Experiments conducted on the benchmark AID dataset demonstrate that the DMFNet achieves an average accuracy of 97.46% $pm$ 0.14%.
  arXiv  Detail & Related papers  (2026-07-16T10:31:32Z)
• Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition [60.20529806857076]
  Multimodal emotion recognition fuses cues such as text, video, and audio to understand individual emotional states.<n>Prior methods face two main limitations: mechanically relying on independent unimodal performance, and coarse-grained fusion conflicting with the fine-grained representations required by emotion tasks.<n>We propose the Complementarity-Supervised Multi-Band Expert Network, named Atsuko, to model fine-grained complementary features via multi-scale band decomposition and expert collaboration.
  arXiv  Detail & Related papers  (2026-03-07T03:58:48Z)
• MM-DETR: An Efficient Multimodal Detection Transformer with Mamba-Driven Dual-Granularity Fusion and Frequency-Aware Modality Adapters [12.063966356953186]
  Multimodal remote sensing object detection aims to achieve more accurate and robust perception under challenging conditions.<n>Existing approaches that rely on attention-based or deformable convolution fusion blocks still struggle to balance performance and lightweight design.<n>We propose MM-DETR, a lightweight and efficient framework for multimodal object detection.
  arXiv  Detail & Related papers  (2025-11-29T07:23:01Z)
• FindRec: Stein-Guided Entropic Flow for Multi-Modal Sequential Recommendation [57.577843653775]
  We propose textbfFindRec (textbfFlexible unified textbfinformation textbfdisentanglement for multi-modal sequential textbfRecommendation)<n>A Stein kernel-based Integrated Information Coordination Module (IICM) theoretically guarantees distribution consistency between multimodal features and ID streams.<n>A cross-modal expert routing mechanism that adaptively filters and combines multimodal features based on their contextual relevance.
  arXiv  Detail & Related papers  (2025-07-07T04:09:45Z)
• Cross-Modal Learning for Anomaly Detection in Complex Industrial Process: Methodology and Benchmark [19.376814754500625]
  Anomaly detection in complex industrial processes plays a pivotal role in ensuring efficient, stable, and secure operation. This paper proposes a cross-modal Transformer to facilitate anomaly detection by exploring the correlation between visual features (video) and process variables (current) in the context of the fused magnesium smelting process. We present a pioneering cross-modal benchmark of the fused magnesium smelting process, featuring synchronously acquired video and current data for over 2.2 million samples.
  arXiv  Detail & Related papers  (2024-06-13T11:40:06Z)
• AMFD: Distillation via Adaptive Multimodal Fusion for Multispectral Pedestrian Detection [23.91870504363899]
  Double-stream networks in multispectral detection employ two separate feature extraction branches for multi-modal data. This has hindered the widespread employment of multispectral pedestrian detection in embedded devices for autonomous systems. We introduce the Adaptive Modal Fusion Distillation (AMFD) framework, which can fully utilize the original modal features of the teacher network.
  arXiv  Detail & Related papers  (2024-05-21T17:17:17Z)
• Fusion-Mamba for Cross-modality Object Detection [63.56296480951342]
  Cross-modality fusing information from different modalities effectively improves object detection performance. We design a Fusion-Mamba block (FMB) to map cross-modal features into a hidden state space for interaction. Our proposed approach outperforms the state-of-the-art methods on $m$AP with 5.9% on $M3FD$ and 4.9% on FLIR-Aligned datasets.
  arXiv  Detail & Related papers  (2024-04-14T05:28:46Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• Object Segmentation by Mining Cross-Modal Semantics [68.88086621181628]
  We propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision.
  arXiv  Detail & Related papers  (2023-05-17T14:30:11Z)
• A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition [7.80238628278552]
  We propose a novel cross-modal fusion network based on self-attention and residual structure (CFN-SR) for multimodal emotion recognition. To verify the effectiveness of the proposed method, we conduct experiments on the RAVDESS dataset. The experimental results show that the proposed CFN-SR achieves the state-of-the-art and obtains 75.76% accuracy with 26.30M parameters.
  arXiv  Detail & Related papers  (2021-11-03T12:24:03Z)
• Deep Multimodal Fusion by Channel Exchanging [87.40768169300898]
  This paper proposes a parameter-free multimodal fusion framework that dynamically exchanges channels between sub-networks of different modalities. The validity of such exchanging process is also guaranteed by sharing convolutional filters yet keeping separate BN layers across modalities, which, as an add-on benefit, allows our multimodal architecture to be almost as compact as a unimodal network.
  arXiv  Detail & Related papers  (2020-11-10T09:53:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.