Shaken or Stirred? An Analysis of MetaFormer's Token Mixing for Medical Imaging

• URL: http://arxiv.org/abs/2510.05971v2
• Date: Mon, 20 Oct 2025 11:01:47 GMT
• Title: Shaken or Stirred? An Analysis of MetaFormer's Token Mixing for Medical Imaging
• Abstract summary: We present the first comprehensive study of token mixers for medical imaging.<n>Our evaluation spans eight datasets covering diverse modalities and common challenges in the medical domain.<n>For segmentation, we find that the local inductive bias of convolutional token mixers is essential.
• Score: 2.8341356563869016
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The generalization of the Transformer architecture via MetaFormer has reshaped our understanding of its success in computer vision. By replacing self-attention with simpler token mixers, MetaFormer provides strong baselines for vision tasks. However, while extensively studied on natural image datasets, its use in medical imaging remains scarce, and existing works rarely compare different token mixers, potentially overlooking more suitable designs choices. In this work, we present the first comprehensive study of token mixers for medical imaging. We systematically analyze pooling-, convolution-, and attention-based token mixers within the MetaFormer architecture on image classification (global prediction task) and semantic segmentation (dense prediction task). Our evaluation spans eight datasets covering diverse modalities and common challenges in the medical domain. Given the prevalence of pretraining from natural images to mitigate medical data scarcity, we also examine transferring pretrained weights to new token mixers. Our results show that, for classification, low-complexity token mixers (e.g. grouped convolution or pooling) are sufficient, aligning with findings on natural images. Pretrained weights remain useful despite the domain gap introduced by the new token mixer. For segmentation, we find that the local inductive bias of convolutional token mixers is essential. Grouped convolutions emerge as the preferred choice, as they reduce runtime and parameter count compared to standard convolutions, while the MetaFormer's channel-MLPs already provide the necessary cross-channel interactions.

Related papers

• Hyperbolic Hierarchical Clustering for Visual Representation Learning [62.38996677789715]
  An effective token mixer is a fundamental component of modern vision backbones like Transformers.<n>We introduce ClusterMixer, a transparent token mixer grounded in a clustering paradigm and interpretable by design.<n>We present HCFormer, a new backbone architecture that integrates ClusterMixer with a series of meticulously designed clustering strategies.
  arXiv  Detail & Related papers  (2026-08-24T00:01:09Z)
• Joint Medical Image Enhancement and Segmentation with Diffusion-based Symbiotic Information Interaction [6.8956820908908965]
  DiSIINet is built on the principle that enhancement and segmentation should mutually reinforce each other in a unified model.<n> Experiments on multi-modal medical datasets (MRI, CT, ultrasound) show that DiSIINet achieves significant performance improvements.
  arXiv  Detail & Related papers  (2026-06-30T07:50:44Z)
• Diffusion-Guided Mask-Consistent Paired Mixing for Endoscopic Image Segmentation [57.37991748282666]
  We propose a paired, diffusion-guided paradigm that fuses the strengths of sample mixing and diffusion synthesis.<n>For each real image, a synthetic counterpart is generated under the same mask and the pair is used as a controllable input for Mask-Consistent Paired Mixing (MCPMix)<n>This produces a continuous family of intermediate samples that smoothly bridges synthetic and real appearances under shared geometry.
  arXiv  Detail & Related papers  (2025-11-05T06:14:19Z)
• SimCroP: Radiograph Representation Learning with Similarity-driven Cross-granularity Pre-training [25.763109982379703]
  We propose a Similarity-Driven Cross-Granularity Pre-training framework on chest CTs.<n>It combines similarity-driven alignment and cross-granularity fusion to improve radiograph interpretation.<n>SimCroP is pre-trained on a large-scale paired CT-reports dataset and validated on image classification and segmentation tasks.
  arXiv  Detail & Related papers  (2025-09-10T06:20:53Z)
• Decompose-and-Compose: A Compositional Approach to Mitigating Spurious Correlation [2.273629240935727]
  We propose Decompose-and-Compose (DaC) to improve correlation shift by combining elements of images. Based on our observations, models trained with Empirical Risk Minimization (ERM) usually highly attend to either the causal components or the components having a high spurious correlation with the label. We propose a group-balancing method by intervening on images without requiring group labels or information regarding the spurious features during training.
  arXiv  Detail & Related papers  (2024-02-29T07:24:24Z)
• Token Fusion: Bridging the Gap between Token Pruning and Token Merging [71.84591084401458]
  Vision Transformers (ViTs) have emerged as powerful backbones in computer vision, outperforming many traditional CNNs. computational overhead, largely attributed to the self-attention mechanism, makes deployment on resource-constrained edge devices challenging. We introduce "Token Fusion" (ToFu), a method that amalgamates the benefits of both token pruning and token merging.
  arXiv  Detail & Related papers  (2023-12-02T04:29:19Z)
• Improving Adversarial Robustness of Masked Autoencoders via Test-time Frequency-domain Prompting [133.55037976429088]
  We investigate the adversarial robustness of vision transformers equipped with BERT pretraining (e.g., BEiT, MAE) A surprising observation is that MAE has significantly worse adversarial robustness than other BERT pretraining methods. We propose a simple yet effective way to boost the adversarial robustness of MAE.
  arXiv  Detail & Related papers  (2023-08-20T16:27:17Z)
• Disruptive Autoencoders: Leveraging Low-level features for 3D Medical Image Pre-training [51.16994853817024]
  This work focuses on designing an effective pre-training framework for 3D radiology images. We introduce Disruptive Autoencoders, a pre-training framework that attempts to reconstruct the original image from disruptions created by a combination of local masking and low-level perturbations. The proposed pre-training framework is tested across multiple downstream tasks and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-07-31T17:59:42Z)
• RIFormer: Keep Your Vision Backbone Effective While Removing Token Mixer [95.71132572688143]
  This paper studies how to keep a vision backbone effective while removing token mixers in its basic building blocks. Token mixers, as self-attention for vision transformers (ViTs), are intended to perform information communication between different spatial tokens but suffer from considerable computational cost and latency.
  arXiv  Detail & Related papers  (2023-04-12T07:34:13Z)
• MetaFormer Baselines for Vision [166.01609889532747]
  We introduce several baseline models under MetaFormer using the most basic or common mixers.<n>We find that MetaFormer ensures solid lower bound of performance.<n>We also find that a new activation, StarReLU, reduces FLOPs of activation compared with GELU yet achieves better performance.
  arXiv  Detail & Related papers  (2022-10-24T17:59:57Z)
• MAGIC: Mask-Guided Image Synthesis by Inverting a Quasi-Robust Classifier [37.774220727662914]
  We propose a one-shot mask-guided image synthesis that allows controlling manipulations of a single image. Our proposed method, entitled MAGIC, leverages structured gradients from a pre-trained quasi-robust classifier. MAGIC aggregates gradients over the input, driven by a guide binary mask that enforces a strong, spatial prior.
  arXiv  Detail & Related papers  (2022-09-23T12:15:40Z)
• TokenMix: Rethinking Image Mixing for Data Augmentation in Vision Transformers [36.630476419392046]
  CutMix is a popular augmentation technique commonly used for training modern convolutional and transformer vision networks. We propose a novel data augmentation technique TokenMix to improve the performance of vision transformers.
  arXiv  Detail & Related papers  (2022-07-18T07:08:29Z)
• CropMix: Sampling a Rich Input Distribution via Multi-Scale Cropping [97.05377757299672]
  We present a simple method, CropMix, for producing a rich input distribution from the original dataset distribution. CropMix can be seamlessly applied to virtually any training recipe and neural network architecture performing classification tasks. We show that CropMix is of benefit to both contrastive learning and masked image modeling towards more powerful representations.
  arXiv  Detail & Related papers  (2022-05-31T16:57:28Z)
• Self Pre-training with Masked Autoencoders for Medical Image Classification and Segmentation [37.25161294917211]
  Masked Autoencoder (MAE) has been shown to be effective in pre-training Vision Transformers (ViT) for natural image analysis. We investigate a self pre-training paradigm with MAE for medical image analysis tasks.
  arXiv  Detail & Related papers  (2022-03-10T16:22:38Z)
• Going to Extremes: Weakly Supervised Medical Image Segmentation [12.700841704699615]
  We suggest using minimal user interaction in the form of extreme point clicks to train a segmentation model. An initial segmentation is generated based on the extreme points utilizing the random walker algorithm. This initial segmentation is then used as a noisy supervision signal to train a fully convolutional network.
  arXiv  Detail & Related papers  (2020-09-25T00:28:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.