DC-ViT: Modulating Spatial and Channel Interactions for Multi-Channel Images

• URL: http://arxiv.org/abs/2603.14281v1
• Date: Sun, 15 Mar 2026 08:31:34 GMT
• Title: DC-ViT: Modulating Spatial and Channel Interactions for Multi-Channel Images
• Abstract summary: Training and evaluation in multi-channel imaging (MCI) remains challenging due to heterogeneous channel configurations.<n>Recent Multi-Channel Vision Transformers (MC-ViTs) address this by enabling flexible channel inputs, typically by jointly encoding patch tokens from all channels within a unified attention space.<n>We propose Decoupled Vision Transformer (DC-ViT), which explicitly regulates information sharing using Decoupled Self-Attention (DSA)
• Score: 9.507520646516719
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training and evaluation in multi-channel imaging (MCI) remains challenging due to heterogeneous channel configurations arising from varying staining protocols, sensor types, and acquisition settings. This heterogeneity limits the applicability of fixed-channel encoders commonly used in general computer vision. Recent Multi-Channel Vision Transformers (MC-ViTs) address this by enabling flexible channel inputs, typically by jointly encoding patch tokens from all channels within a unified attention space. However, unrestricted token interactions across channels can lead to feature dilution, reducing the ability to preserve channel-specific semantics that are critical in MCI data. To address this, we propose Decoupled Vision Transformer (DC-ViT), which explicitly regulates information sharing using Decoupled Self-Attention (DSA), which decomposes token updates into two complementary pathways: spatial updates that model intra-channel structure, and channel-wise updates that adaptively integrate cross-channel information. This decoupling mitigates informational collapse while allowing selective inter-channel interaction. To further exploit these enhanced channel-specific representations, we introduce Decoupled Aggregation (DAG), which allows the model to learn task-specific channel importances. Extensive experiments across three MCI benchmarks demonstrate consistent improvements over existing MC-ViT approaches.

Related papers

• Channel-Free Human Activity Recognition via Inductive-Bias-Aware Fusion Design for Heterogeneous IoT Sensor Environments [0.0]
  Human activity recognition (HAR) in Internet of Things (IoT) environments must cope with heterogeneous sensor settings.<n>We investigate strict channel-free HAR, in which a single shared model performs inference without assuming a fixed number, order, or semantic arrangement of input channels.<n>We propose a channel-free HAR framework that combines channel-wise encoding with a shared encoder, metadata-conditioned late fusion, and joint optimization of channel-level and fused predictions.
  arXiv  Detail & Related papers  (2026-04-23T07:32:36Z)
• CI-ICM: Channel Importance-driven Learned Image Coding for Machines [56.1188469769241]
  We propose a Channel Importance-driven learned Image Coding for Machines (CI-ICM) to maximize the performance of machine vision tasks.<n>The proposed CI-ICM achieves BD-mAP@50:95 gains of 16.25%$ in object detection and 13.72%$ in instance segmentation over the established baseline.
  arXiv  Detail & Related papers  (2026-04-07T02:36:47Z)
• Routing Channel-Patch Dependencies in Time Series Forecasting with Graph Spectral Decomposition [44.75749696286735]
  We propose a generic plugin xCPD that can adaptively model the channel-patch dependencies from the perspective of graph spectral decomposition.<n>xCPD can be seamlessly integrated on top of existing CI and CD forecasting models, consistently enhancing both accuracy and generalization across benchmarks.
  arXiv  Detail & Related papers  (2026-03-14T02:18:17Z)
• Context Video Semantic Transmission with Variable Length and Rate Coding over MIMO Channels [49.624608869195065]
  We propose the context video semantic transmission (CVST) framework for wireless video transmission.<n>We learn a context-channel correlation map to explicitly formulate the relationships between feature groups and multiple input multiple output (MIMO) subchannels.<n>We demonstrate substantial performance gains over various standardized separated coding methods and recent wireless video semantic communication approaches.
  arXiv  Detail & Related papers  (2025-12-23T10:48:43Z)
• ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning [27.124745103676734]
  ChA-MAEViT is an MAE-based method that enhances feature learning across Multi-Channel Imaging channels.<n>We show that ChA-MAEViT significantly outperforms state-of-the-art MCI-ViTs by 3.0-21.5%.
  arXiv  Detail & Related papers  (2025-03-25T03:45:59Z)
• Isolated Channel Vision Transformers: From Single-Channel Pretraining to Multi-Channel Finetuning [4.376574371216863]
  We introduce a simple yet effective pretraining framework for large-scale MCI datasets.<n>Our method, named Isolated Channel ViT (IC-ViT), patchifies image channels individually and thereby enables pretraining for multimodal multi-channel tasks.<n> Experiments on various tasks and benchmarks, including JUMP-CP and CHAMMI for cell microscopy imaging, and So2Sat-LCZ42 for satellite imaging, show that the proposed IC-ViT delivers 4-14 percentage points of performance improvement.
  arXiv  Detail & Related papers  (2025-03-12T20:45:02Z)
• InjectTST: A Transformer Method of Injecting Global Information into Independent Channels for Long Time Series Forecasting [28.138548907887667]
  We propose an injection method for global information into channel-independent Transformer, InjectTST. Experiments indicate that InjectTST can achieve stable improvement compared with state-of-the-art models.
  arXiv  Detail & Related papers  (2024-03-05T09:33:36Z)
• SCTransNet: Spatial-channel Cross Transformer Network for Infrared Small Target Detection [46.049401912285134]
  Infrared small target detection (IRSTD) has recently benefitted greatly from U-shaped neural models. Existing techniques struggle when the target has high similarities with the background. We present a Spatial-channel Cross Transformer Network (SCTransNet) that leverages spatial-channel cross transformer blocks.
  arXiv  Detail & Related papers  (2024-01-28T06:41:15Z)
• Joint Channel Estimation and Feedback with Masked Token Transformers in Massive MIMO Systems [74.52117784544758]
  This paper proposes an encoder-decoder based network that unveils the intrinsic frequency-domain correlation within the CSI matrix. The entire encoder-decoder network is utilized for channel compression. Our method outperforms state-of-the-art channel estimation and feedback techniques in joint tasks.
  arXiv  Detail & Related papers  (2023-06-08T06:15:17Z)
• Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets [91.25055890980084]
  There still remains an extreme performance gap between Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) when training from scratch on small datasets. We propose Dynamic Hybrid Vision Transformer (DHVT) as the solution to enhance the two inductive biases. Our DHVT achieves a series of state-of-the-art performance with a lightweight model, 85.68% on CIFAR-100 with 22.8M parameters, 82.3% on ImageNet-1K with 24.0M parameters.
  arXiv  Detail & Related papers  (2022-10-12T06:54:39Z)
• Volumetric Transformer Networks [88.85542905676712]
  We introduce a learnable module, the volumetric transformer network (VTN) VTN predicts channel-wise warping fields so as to reconfigure intermediate CNN features spatially and channel-wisely. Our experiments show that VTN consistently boosts the features' representation power and consequently the networks' accuracy on fine-grained image recognition and instance-level image retrieval.
  arXiv  Detail & Related papers  (2020-07-18T14:00:12Z)
• Channel Interaction Networks for Fine-Grained Image Categorization [61.095320862647476]
  Fine-grained image categorization is challenging due to the subtle inter-class differences. We propose a channel interaction network (CIN), which models the channel-wise interplay both within an image and across images. Our model can be trained efficiently in an end-to-end fashion without the need of multi-stage training and testing.
  arXiv  Detail & Related papers  (2020-03-11T11:51:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.