Robust Representation Learning in Masked Autoencoders

• URL: http://arxiv.org/abs/2602.03531v1
• Date: Tue, 03 Feb 2026 13:48:34 GMT
• Title: Robust Representation Learning in Masked Autoencoders
• Authors: Anika Shrivastava, Renu Rameshan, Samar Agnihotri,
• Abstract summary: Masked Autoencoders (MAEs) achieve impressive performance in image classification tasks, yet the internal representations they learn remain less understood.<n>This work started as an attempt to understand the strong downstream classification performance of MAE.
• Score: 2.599882743586164
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Masked Autoencoders (MAEs) achieve impressive performance in image classification tasks, yet the internal representations they learn remain less understood. This work started as an attempt to understand the strong downstream classification performance of MAE. In this process we discover that representations learned with the pretraining and fine-tuning, are quite robust - demonstrating a good classification performance in the presence of degradations, such as blur and occlusions. Through layer-wise analysis of token embeddings, we show that pretrained MAE progressively constructs its latent space in a class-aware manner across network depth: embeddings from different classes lie in subspaces that become increasingly separable. We further observe that MAE exhibits early and persistent global attention across encoder layers, in contrast to standard Vision Transformers (ViTs). To quantify feature robustness, we introduce two sensitivity indicators: directional alignment between clean and perturbed embeddings, and head-wise retention of active features under degradations. These studies help establish the robust classification performance of MAEs.

Related papers

• Revealing the Attention Floating Mechanism in Masked Diffusion Models [52.74142815156738]
  Masked diffusion models (MDMs) leverage bidirectional attention and a denoising process.<n>This paper investigates the attention behaviors in MDMs, revealing the phenomenon of Attention Floating.
  arXiv  Detail & Related papers  (2026-01-12T09:10:05Z)
• MTS-DMAE: Dual-Masked Autoencoder for Unsupervised Multivariate Time Series Representation Learning [39.592562986835595]
  We propose Dual-Masked Autoencoder (DMAE) for unsupervised MTS representation learning.<n>DMAE formulates two complementary pretext tasks: (1) reconstructing masked values based on visible attributes, and (2) estimating latent representations of masked features, guided by a teacher encoder.<n>By jointly optimizing these objectives, DMAE learns temporally coherent and semantically rich representations.
  arXiv  Detail & Related papers  (2025-09-19T15:25:43Z)
• Mask & Match: Learning to Recognize Handwritten Math with Self-Supervised Attention [0.19116784879310025]
  We present a self-supervised learning framework for recognizing handwritten mathematical expressions (HMER)<n>Our approach begins by pretraining an image encoder using a combination of global and local contrastive loss.<n>A key contribution of this work is a novel self-supervised attention network, which is trained using a progressive spatial masking strategy.
  arXiv  Detail & Related papers  (2025-08-08T08:11:36Z)
• Semi-supervised Semantic Segmentation with Multi-Constraint Consistency Learning [81.02648336552421]
  We propose a Multi-Constraint Consistency Learning approach to facilitate the staged enhancement of the encoder and decoder.<n>Self-adaptive feature masking and noise injection are designed in an instance-specific manner to perturb the features for robust learning of the decoder.<n> Experimental results on Pascal VOC2012 and Cityscapes datasets demonstrate that our proposed MCCL achieves new state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-03-23T03:21:33Z)
• Bringing Masked Autoencoders Explicit Contrastive Properties for Point Cloud Self-Supervised Learning [116.75939193785143]
  Contrastive learning (CL) for Vision Transformers (ViTs) in image domains has achieved performance comparable to CL for traditional convolutional backbones. In 3D point cloud pretraining with ViTs, masked autoencoder (MAE) modeling remains dominant.
  arXiv  Detail & Related papers  (2024-07-08T12:28:56Z)
• Learning from Mistakes: Self-Regularizing Hierarchical Representations in Point Cloud Semantic Segmentation [15.353256018248103]
  LiDAR semantic segmentation has gained attention to accomplish fine-grained scene understanding. We present a coarse-to-fine setup that LEArns from classification mistaKes (LEAK) derived from a standard model. Our LEAK approach is very general and can be seamlessly applied on top of any segmentation architecture.
  arXiv  Detail & Related papers  (2023-01-26T14:52:30Z)
• i-MAE: Are Latent Representations in Masked Autoencoders Linearly Separable? [26.146459754995597]
  Masked image modeling (MIM) has been recognized as a strong self-supervised pre-training approach in the vision domain. This paper aims to explore an interactive Masked Autoencoders (i-MAE) framework to enhance the representation capability. In addition to qualitatively analyzing the characteristics of the latent representations, we examine the existence of linear separability and the degree of semantics in the latent space.
  arXiv  Detail & Related papers  (2022-10-20T17:59:54Z)
• MAPLE: Masked Pseudo-Labeling autoEncoder for Semi-supervised Point Cloud Action Recognition [160.49403075559158]
  We propose a Masked Pseudo-Labeling autoEncoder (textbfMAPLE) framework for point cloud action recognition. In particular, we design a novel and efficient textbfDecoupled textbfspatial-textbftemporal TranstextbfFormer (textbfDestFormer) as the backbone of MAPLE. MAPLE achieves superior results on three public benchmarks and outperforms the state-of-the-art method by 8.08% accuracy on the MSR-Action3
  arXiv  Detail & Related papers  (2022-09-01T12:32:40Z)
• SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption [72.35532598131176]
  We propose SCARF, a technique for contrastive learning, where views are formed by corrupting a random subset of features. We show that SCARF complements existing strategies and outperforms alternatives like autoencoders.
  arXiv  Detail & Related papers  (2021-06-29T08:08:33Z)
• Can Semantic Labels Assist Self-Supervised Visual Representation Learning? [194.1681088693248]
  We present a new algorithm named Supervised Contrastive Adjustment in Neighborhood (SCAN) In a series of downstream tasks, SCAN achieves superior performance compared to previous fully-supervised and self-supervised methods. Our study reveals that semantic labels are useful in assisting self-supervised methods, opening a new direction for the community.
  arXiv  Detail & Related papers  (2020-11-17T13:25:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.