Co-distilled attention guided masked image modeling with noisy teacher for self-supervised learning on medical images

• URL: http://arxiv.org/abs/2604.14506v1
• Date: Thu, 16 Apr 2026 00:43:39 GMT
• Title: Co-distilled attention guided masked image modeling with noisy teacher for self-supervised learning on medical images
• Abstract summary: Masked image modeling (MIM) is a highly effective self-supervised learning (SSL) approach to extract useful feature representations from unannotated data.<n>We introduce an attention guided masking mechanism for Swin within a co-distillation learning framework to selectively mask semantically co-occurring and discriminative patches.<n>We demonstrate the capability of DAGMaN on multiple tasks including full- and few-shot lung nodule classification, immunotherapy outcome prediction, tumor segmentation, and unsupervised organs clustering.
• Score: 5.248922934004839
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Masked image modeling (MIM) is a highly effective self-supervised learning (SSL) approach to extract useful feature representations from unannotated data. Predominantly used random masking methods make SSL less effective for medical images due to the contextual similarity of neighboring patches, leading to information leakage and SSL simplification. Hierarchical shifted window (Swin) transformer, a highly effective approach for medical images cannot use advanced masking methods as it lacks a global [CLS] token. Hence, we introduced an attention guided masking mechanism for Swin within a co-distillation learning framework to selectively mask semantically co-occurring and discriminative patches, to reduce information leakage and increase the difficulty of SSL pretraining. However, attention guided masking inevitably reduces the diversity of attention heads, which negatively impacts downstream task performance. To address this, we for the first time, integrate a noisy teacher into the co-distillation framework (termed DAGMaN) that performs attentive masking while preserving high attention head diversity. We demonstrate the capability of DAGMaN on multiple tasks including full- and few-shot lung nodule classification, immunotherapy outcome prediction, tumor segmentation, and unsupervised organs clustering.

Related papers

• MIRAM: Masked Image Reconstruction Across Multiple Scales for Breast Lesion Risk Prediction [2.0199924721373392]
  Masked image modeling (MIM) has emerged as a more potent SSL technique.<n>This research paper introduces a scalable and practical SSL approach centered around more challenging pretext tasks.<n>Our hypothesis posits that reconstructing high-resolution images enables the model to attend to finer spatial details.
  arXiv  Detail & Related papers  (2025-03-10T10:32:55Z)
• Enhanced Masked Image Modeling to Avoid Model Collapse on Multi-modal MRI Datasets [6.3467517115551875]
  Masked image modeling (MIM) has shown promise in utilizing unlabeled data.<n>We analyze and address model collapse in two types: complete collapse and dimensional collapse.<n>We construct the enhanced MIM (E-MIM) with HMP and PBT module to avoid model collapse multi-modal MRI.
  arXiv  Detail & Related papers  (2024-07-15T01:11:30Z)
• AnatoMask: Enhancing Medical Image Segmentation with Reconstruction-guided Self-masking [5.844539603252746]
  Masked image modeling (MIM) has shown effectiveness by reconstructing randomly masked images to learn detailed representations. We propose AnatoMask, a novel MIM method that leverages reconstruction loss to dynamically identify and mask out anatomically significant regions.
  arXiv  Detail & Related papers  (2024-07-09T00:15:52Z)
• Self-distilled Masked Attention guided masked image modeling with noise Regularized Teacher (SMART) for medical image analysis [6.712251433139412]
  Pretraining vision transformers (ViT) with attention guided masked image modeling (MIM) has shown to increase downstream accuracy for natural image analysis. We developed a co-distilled Swin transformer that combines a noisy momentum updated teacher to guide selective masking for MIM.
  arXiv  Detail & Related papers  (2023-10-02T13:53:55Z)
• Masking Improves Contrastive Self-Supervised Learning for ConvNets, and Saliency Tells You Where [63.61248884015162]
  We aim to alleviate the burden of including masking operation into the contrastive-learning framework for convolutional neural networks. We propose to explicitly take the saliency constraint into consideration in which the masked regions are more evenly distributed among the foreground and background.
  arXiv  Detail & Related papers  (2023-09-22T09:58:38Z)
• AMLP:Adaptive Masking Lesion Patches for Self-supervised Medical Image Segmentation [67.97926983664676]
  Self-supervised masked image modeling has shown promising results on natural images. However, directly applying such methods to medical images remains challenging. We propose a novel self-supervised medical image segmentation framework, Adaptive Masking Lesion Patches (AMLP)
  arXiv  Detail & Related papers  (2023-09-08T13:18:10Z)
• Disruptive Autoencoders: Leveraging Low-level features for 3D Medical Image Pre-training [51.16994853817024]
  This work focuses on designing an effective pre-training framework for 3D radiology images. We introduce Disruptive Autoencoders, a pre-training framework that attempts to reconstruct the original image from disruptions created by a combination of local masking and low-level perturbations. The proposed pre-training framework is tested across multiple downstream tasks and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-07-31T17:59:42Z)
• Improving Masked Autoencoders by Learning Where to Mask [65.89510231743692]
  Masked image modeling is a promising self-supervised learning method for visual data. We present AutoMAE, a framework that uses Gumbel-Softmax to interlink an adversarially-trained mask generator and a mask-guided image modeling process. In our experiments, AutoMAE is shown to provide effective pretraining models on standard self-supervised benchmarks and downstream tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:28:55Z)
• MPS-AMS: Masked Patches Selection and Adaptive Masking Strategy Based Self-Supervised Medical Image Segmentation [46.76171191827165]
  We propose masked patches selection and adaptive masking strategy based self-supervised medical image segmentation method, named MPS-AMS. Our proposed method greatly outperforms the state-of-the-art self-supervised baselines.
  arXiv  Detail & Related papers  (2023-02-27T11:57:06Z)
• A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond [64.85076239939336]
  Self-supervised learning (SSL) in vision might undertake a similar trajectory as in NLP. generative pretext tasks with the masked prediction (e.g., BERT) have become a de facto standard SSL practice in NLP. Success of mask image modeling has revived the masking autoencoder.
  arXiv  Detail & Related papers  (2022-07-30T09:59:28Z)
• Adversarial Masking for Self-Supervised Learning [81.25999058340997]
  Masked image model (MIM) framework for self-supervised learning, ADIOS, is proposed. It simultaneously learns a masking function and an image encoder using an adversarial objective. It consistently improves on state-of-the-art self-supervised learning (SSL) methods on a variety of tasks and datasets.
  arXiv  Detail & Related papers  (2022-01-31T10:23:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.