A Dual-Modulation Framework for RGB-T Crowd Counting via Spatially Modulated Attention and Adaptive Fusion

• URL: http://arxiv.org/abs/2509.17079v1
• Date: Sun, 21 Sep 2025 13:52:29 GMT
• Title: A Dual-Modulation Framework for RGB-T Crowd Counting via Spatially Modulated Attention and Adaptive Fusion
• Abstract summary: We propose a Dual Modulation Framework for accurate RGB-Thermal (RGB-T) crowd counting.<n>We use two modules: Spatially Modulated Attention (SMA) and Adaptive Fusion Modulation (AFM)<n>Experiments on RGB-T crowd counting datasets demonstrate the superior performance of our method compared to previous works.
• Score: 10.919439739345195
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurate RGB-Thermal (RGB-T) crowd counting is crucial for public safety in challenging conditions. While recent Transformer-based methods excel at capturing global context, their inherent lack of spatial inductive bias causes attention to spread to irrelevant background regions, compromising crowd localization precision. Furthermore, effectively bridging the gap between these distinct modalities remains a major hurdle. To tackle this, we propose the Dual Modulation Framework, comprising two modules: Spatially Modulated Attention (SMA), which improves crowd localization by using a learnable Spatial Decay Mask to penalize attention between distant tokens and prevent focus from spreading to the background; and Adaptive Fusion Modulation (AFM), which implements a dynamic gating mechanism to prioritize the most reliable modality for adaptive cross-modal fusion. Extensive experiments on RGB-T crowd counting datasets demonstrate the superior performance of our method compared to previous works. Code available at https://github.com/Cht2924/RGBT-Crowd-Counting.

Related papers

• CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detection [17.487415888287]
  Event cameras capture sparse brightness changes with high temporal resolution and high dynamic range.<n>Previous multi-modal fusion techniques fail to handle the inherent heterogeneity between RGB frames and event streams.<n>We propose a Cross-Modal information inTeraction transFormer, which hierarchically integrates RGB and event information.
  arXiv  Detail & Related papers  (2026-06-28T00:49:05Z)
• Active Adversarial Perturbation-driven Associative Memory Retrieval for RGB-Event Visual Object Tracking [58.338848435042536]
  RGB-Event tracking improves localization by fusing RGB appearance textures and dense temporal motion cues from event sensors.<n>Real-world scenes suffer diverse structured signal degradations that hinder traditional multi-modal fusion.<n>We present a hierarchical perturbation and retrieval framework tailored for RGB-Event tracking with robustness against partial target missing and modal degradation.
  arXiv  Detail & Related papers  (2026-06-24T23:34:06Z)
• Frequency-Domain Regularized Adversarial Alignment for Transferable Attacks against Closed-Source MLLMs [59.973833105841685]
  A key challenge for improving adversarial transferability is to effectively capture the intrinsic visual focus shared across different models.<n>We propose FRA-Attack, which addresses both challenges from a unified frequency-domain regularization perspective.<n>For feature alignment, a high-pass DCT objective on patch features suppresses redundant global structures and concentrates the loss on the high-frequency band that carries the MLLMs' intrinsic visual focus.<n>For gradient optimization, we introduce Frequency-domain Gradient Regularization (FGR), a textitmodel-agnostic low-pass regularizer that modulates the surrogate gradient using
  arXiv  Detail & Related papers  (2026-05-20T08:15:56Z)
• RACANet: Reliability-Aware Crowd Anchor Network for RGB-T Crowd Counting [15.536708520170095]
  RGB-Thermal (T) crowd counting aims to integrate visible-spectrum and thermal infrared information to improve the robustness of crowd density estimation in complex scenes.<n>This paper proposes a two-stage fusion framework, RACANet, a Reliability-Aware Crowd Anchor Network for RGB-T crowd counting.<n>Experiments conducted on two widely used benchmark datasets, RGBT-CC and Drone-RGBT, demonstrate that RACANet outperforms existing methods.
  arXiv  Detail & Related papers  (2026-04-27T14:38:49Z)
• RTFDNet: Fusion-Decoupling for Robust RGB-T Segmentation [0.2578242050187029]
  RGB-Thermal (RGB-T) semantic segmentation is essential for robotic systems operating in low-light environments.<n>Traditional approaches often overemphasize modality balance, resulting in limited robustness or severe performance when sensor signals are partially missing.
  arXiv  Detail & Related papers  (2026-03-10T03:40:26Z)
• Rethinking Multi-Condition DiTs: Eliminating Redundant Attention via Position-Alignment and Keyword-Scoping [61.459927600301654]
  Multi-condition control is bottlenecked by the conventional concatenate-and-attend'' strategy.<n>Our analysis reveals that much of this cross-modal interaction is spatially or semantically redundant.<n>We propose Position-aligned and Keyword-scoped Attention (PKA), a highly efficient framework designed to eliminate these redundancies.
  arXiv  Detail & Related papers  (2026-02-06T16:39:10Z)
• PEPR: Privileged Event-based Predictive Regularization for Domain Generalization [19.185122873391517]
  We propose a cross-modal framework under the learning using privileged information (LUPI) paradigm for training a robust, single-modality RGB model.<n>We leverage event cameras as a source of privileged information, available only during training.<n>We train the RGB encoder with PEPR to predict event-based latent features, distilling robustness without sacrificing semantic richness.
  arXiv  Detail & Related papers  (2026-02-04T14:10:36Z)
• LEAF-Mamba: Local Emphatic and Adaptive Fusion State Space Model for RGB-D Salient Object Detection [31.453313049462718]
  RGB-D salient object detection (SOD) aims to identify the most conspicuous objects in a scene with the incorporation of depth cues.<n>Existing methods mainly rely on CNNs, limited by the local receptive fields, or Vision Transformers that suffer from the cost of quadratic complexity.<n>We propose a Local Emphatic and Adaptive Fusion state space model (LEAF-Mamba) that contains two novel components.
  arXiv  Detail & Related papers  (2025-09-23T06:08:17Z)
• Transformer-Based Dual-Optical Attention Fusion Crowd Head Point Counting and Localization Network [9.214772627896156]
  The model designs a dual-optical attention fusion module (DAFP) by introducing complementary information from infrared images.<n>The proposed method outperforms existing techniques in terms of performance, especially in challenging dense low-light scenes.
  arXiv  Detail & Related papers  (2025-05-11T10:55:14Z)
• FUSE: Label-Free Image-Event Joint Monocular Depth Estimation via Frequency-Decoupled Alignment and Degradation-Robust Fusion [63.87313550399871]
  Image-event joint depth estimation methods leverage complementary modalities for robust perception, yet face challenges in generalizability.<n>We propose Self-supervised Transfer (PST) and FrequencyDe-coupled Fusion module (FreDF)<n>PST establishes cross-modal knowledge transfer through latent space alignment with image foundation models.<n>FreDF explicitly decouples high-frequency edge features from low-frequency structural components, resolving modality-specific frequency mismatches.
  arXiv  Detail & Related papers  (2025-03-25T15:04:53Z)
• Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation [54.96563068182733]
  We propose Modality Adaptation with text-to-image Diffusion Models (MADM) for semantic segmentation task. MADM utilizes text-to-image diffusion models pre-trained on extensive image-text pairs to enhance the model's cross-modality capabilities. We show that MADM achieves state-of-the-art adaptation performance across various modality tasks, including images to depth, infrared, and event modalities.
  arXiv  Detail & Related papers  (2024-10-29T03:49:40Z)
• RGB-T Tracking Based on Mixed Attention [5.151994214135177]
  RGB-T tracking involves the use of images from both visible and thermal modalities. An RGB-T tracker based on mixed attention mechanism to achieve complementary fusion of modalities is proposed in this paper.
  arXiv  Detail & Related papers  (2023-04-09T15:59:41Z)
• MAFNet: A Multi-Attention Fusion Network for RGB-T Crowd Counting [40.4816930622052]
  We propose a two-stream RGB-T crowd counting network called Multi-Attention Fusion Network (MAFNet) In the encoder part, a Multi-Attention Fusion (MAF) module is embedded into different stages of the two modality-specific branches for cross-modal fusion. Extensive experiments on two popular datasets show that the proposed MAFNet is effective for RGB-T crowd counting.
  arXiv  Detail & Related papers  (2022-08-14T02:42:09Z)
• Dual Swin-Transformer based Mutual Interactive Network for RGB-D Salient Object Detection [67.33924278729903]
  In this work, we propose Dual Swin-Transformer based Mutual Interactive Network. We adopt Swin-Transformer as the feature extractor for both RGB and depth modality to model the long-range dependencies in visual inputs. Comprehensive experiments on five standard RGB-D SOD benchmark datasets demonstrate the superiority of the proposed DTMINet method.
  arXiv  Detail & Related papers  (2022-06-07T08:35:41Z)
• Transformer-based Network for RGB-D Saliency Detection [82.6665619584628]
  Key to RGB-D saliency detection is to fully mine and fuse information at multiple scales across the two modalities. We show that transformer is a uniform operation which presents great efficacy in both feature fusion and feature enhancement. Our proposed network performs favorably against state-of-the-art RGB-D saliency detection methods.
  arXiv  Detail & Related papers  (2021-12-01T15:53:58Z)
• RGB-D Salient Object Detection with Cross-Modality Modulation and Selection [126.4462739820643]
  We present an effective method to progressively integrate and refine the cross-modality complementarities for RGB-D salient object detection (SOD) The proposed network mainly solves two challenging issues: 1) how to effectively integrate the complementary information from RGB image and its corresponding depth map, and 2) how to adaptively select more saliency-related features.
  arXiv  Detail & Related papers  (2020-07-14T14:22:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.