Sparse Mixture-of-Experts for Multi-Channel Imaging: Are All Channel Interactions Required?

• URL: http://arxiv.org/abs/2511.17400v1
• Date: Fri, 21 Nov 2025 17:00:02 GMT
• Title: Sparse Mixture-of-Experts for Multi-Channel Imaging: Are All Channel Interactions Required?
• Abstract summary: MoE-ViT is a Mixture-of-Experts architecture for multi-channel images in $textViTs$.<n>It treats each channel as an expert and employs a lightweight router to select only the most relevant experts per patch for attention.<n>It achieves substantial efficiency gains without sacrificing, and in some cases enhancing, performance.
• Score: 10.223386883838186
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision Transformers ($\text{ViTs}$) have become the backbone of vision foundation models, yet their optimization for multi-channel domains - such as cell painting or satellite imagery - remains underexplored. A key challenge in these domains is capturing interactions between channels, as each channel carries different information. While existing works have shown efficacy by treating each channel independently during tokenization, this approach naturally introduces a major computational bottleneck in the attention block - channel-wise comparisons leads to a quadratic growth in attention, resulting in excessive $\text{FLOPs}$ and high training cost. In this work, we shift focus from efficacy to the overlooked efficiency challenge in cross-channel attention and ask: "Is it necessary to model all channel interactions?". Inspired by the philosophy of Sparse Mixture-of-Experts ($\text{MoE}$), we propose MoE-ViT, a Mixture-of-Experts architecture for multi-channel images in $\text{ViTs}$, which treats each channel as an expert and employs a lightweight router to select only the most relevant experts per patch for attention. Proof-of-concept experiments on real-world datasets - JUMP-CP and So2Sat - demonstrate that $\text{MoE-ViT}$ achieves substantial efficiency gains without sacrificing, and in some cases enhancing, performance, making it a practical and attractive backbone for multi-channel imaging.

Related papers

• ChannelTok: Efficient Flexible-Length Vision Tokenization [50.67214721388959]
  We introduce a simple, lightweight, and fast channel-wise flexible-length tokenizer.<n>Our method treats each latent channel as a visual token, enabling a parameter-efficient CNN-Transformer hybrid backbone.<n>We validate our approach through extensive experiments on ImageNet, demonstrating consistent quality across diverse token budgets.
  arXiv  Detail & Related papers  (2026-06-03T05:10:51Z)
• DC-ViT: Modulating Spatial and Channel Interactions for Multi-Channel Images [9.507520646516719]
  Training and evaluation in multi-channel imaging (MCI) remains challenging due to heterogeneous channel configurations.<n>Recent Multi-Channel Vision Transformers (MC-ViTs) address this by enabling flexible channel inputs, typically by jointly encoding patch tokens from all channels within a unified attention space.<n>We propose Decoupled Vision Transformer (DC-ViT), which explicitly regulates information sharing using Decoupled Self-Attention (DSA)
  arXiv  Detail & Related papers  (2026-03-15T08:31:34Z)
• ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning [27.124745103676734]
  ChA-MAEViT is an MAE-based method that enhances feature learning across Multi-Channel Imaging channels.<n>We show that ChA-MAEViT significantly outperforms state-of-the-art MCI-ViTs by 3.0-21.5%.
  arXiv  Detail & Related papers  (2025-03-25T03:45:59Z)
• Isolated Channel Vision Transformers: From Single-Channel Pretraining to Multi-Channel Finetuning [4.376574371216863]
  We introduce a simple yet effective pretraining framework for large-scale MCI datasets.<n>Our method, named Isolated Channel ViT (IC-ViT), patchifies image channels individually and thereby enables pretraining for multimodal multi-channel tasks.<n> Experiments on various tasks and benchmarks, including JUMP-CP and CHAMMI for cell microscopy imaging, and So2Sat-LCZ42 for satellite imaging, show that the proposed IC-ViT delivers 4-14 percentage points of performance improvement.
  arXiv  Detail & Related papers  (2025-03-12T20:45:02Z)
• Distilling Channels for Efficient Deep Tracking [68.13422829310835]
  This paper presents a novel framework termed channel distillation to facilitate deep trackers. We show that an integrated formulation can turn feature compression, response map generation, and model update into a unified energy minimization problem. The resulting deep tracker is accurate, fast, and has low memory requirements.
  arXiv  Detail & Related papers  (2024-09-18T08:09:20Z)
• (PASS) Visual Prompt Locates Good Structure Sparsity through a Recurrent HyperNetwork [60.889175951038496]
  Large-scale neural networks have demonstrated remarkable performance in different domains like vision and language processing. One of the key questions of structural pruning is how to estimate the channel significance. We propose a novel algorithmic framework, namely textttPASS. It is a tailored hyper-network to take both visual prompts and network weight statistics as input, and output layer-wise channel sparsity in a recurrent manner.
  arXiv  Detail & Related papers  (2024-07-24T16:47:45Z)
• SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusion [59.96233305733875]
  Time series forecasting plays a crucial role in various fields such as finance, traffic management, energy, and healthcare. Several methods utilize mechanisms like attention or mixer to address this by capturing channel correlations. This paper presents an efficient-based model, the Series-cOre Fused Time Series forecaster (SOFTS)
  arXiv  Detail & Related papers  (2024-04-22T14:06:35Z)
• Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words [7.210982964205077]
  Vision Transformer (ViT) has emerged as a powerful architecture in modern computer vision. However, its application in certain imaging fields, such as microscopy and satellite imaging, presents unique challenges. We propose a modification to the ViT architecture that enhances reasoning across the input channels.
  arXiv  Detail & Related papers  (2023-09-28T02:20:59Z)
• Efficient Multi-Scale Attention Module with Cross-Spatial Learning [4.046170185945849]
  A novel efficient multi-scale attention (EMA) module is proposed. We focus on retaining the information on per channel and decreasing the computational overhead. We conduct extensive ablation studies and experiments on image classification and object detection tasks.
  arXiv  Detail & Related papers  (2023-05-23T00:35:47Z)
• Channel-wise Knowledge Distillation for Dense Prediction [73.99057249472735]
  We propose to align features channel-wise between the student and teacher networks. We consistently achieve superior performance on three benchmarks with various network structures.
  arXiv  Detail & Related papers  (2020-11-26T12:00:38Z)
• Operation-Aware Soft Channel Pruning using Differentiable Masks [51.04085547997066]
  We propose a data-driven algorithm, which compresses deep neural networks in a differentiable way by exploiting the characteristics of operations. We perform extensive experiments and achieve outstanding performance in terms of the accuracy of output networks.
  arXiv  Detail & Related papers  (2020-07-08T07:44:00Z)
• Channel Interaction Networks for Fine-Grained Image Categorization [61.095320862647476]
  Fine-grained image categorization is challenging due to the subtle inter-class differences. We propose a channel interaction network (CIN), which models the channel-wise interplay both within an image and across images. Our model can be trained efficiently in an end-to-end fashion without the need of multi-stage training and testing.
  arXiv  Detail & Related papers  (2020-03-11T11:51:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.