Less is More: Decoder-Free Masked Modeling for Efficient Skeleton Representation Learning

• URL: http://arxiv.org/abs/2603.10648v2
• Date: Thu, 12 Mar 2026 04:01:43 GMT
• Title: Less is More: Decoder-Free Masked Modeling for Efficient Skeleton Representation Learning
• Abstract summary: Skeleton-based action representation learning has evolved from Contrastive Learning (CL) to Masked Auto-Encoder (MAE)<n>We propose SLiM, a novel unified framework that harmonizes masked modeling with contrastive learning via a shared encoder.<n>We show that SLiM consistently achieves state-of-the-art performance across all downstream protocols.
• Score: 28.87004127483584
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The landscape of skeleton-based action representation learning has evolved from Contrastive Learning (CL) to Masked Auto-Encoder (MAE) architectures. However, each paradigm faces inherent limitations: CL often overlooks fine-grained local details, while MAE is burdened by computationally heavy decoders. Moreover, MAE suffers from severe computational asymmetry -- benefiting from efficient masking during pre-training but requiring exhaustive full-sequence processing for downstream tasks. To resolve these bottlenecks, we propose SLiM (Skeleton Less is More), a novel unified framework that harmonizes masked modeling with contrastive learning via a shared encoder. By eschewing the reconstruction decoder, SLiM not only eliminates computational redundancy but also compels the encoder to capture discriminative features directly. SLiM is the first framework with decoder-free masked modeling of representative learning. Crucially, to prevent trivial reconstruction arising from high skeletal-temporal correlation, we introduce semantic tube masking, alongside skeletal-aware augmentations designed to ensure anatomical consistency across diverse temporal granularities. Extensive experiments demonstrate that SLiM consistently achieves state-of-the-art performance across all downstream protocols. Notably, our method delivers this superior accuracy with exceptional efficiency, reducing inference computational cost by 7.89x compared to existing MAE methods.

Related papers

• SAMIC: A Lightweight Semantic-Aware Mamba for Efficient Perceptual Image Compression [5.394151637957463]
  We present an efficient perceptual image compression method that exploits the long-range modeling capability and linear computational complexity of state space models.<n>We develop a semantic-aware Mamba block (SAMB) to enable scanning guided by dynamically clustered semantic features.<n>Our method performs favorably against state-of-the-art approaches in terms of rate-distortion-perception tradeoff and model complexity.
  arXiv  Detail & Related papers  (2026-05-06T07:03:04Z)
• Towards Efficient General Feature Prediction in Masked Skeleton Modeling [59.46799426434277]
  We propose a novel General Feature Prediction framework (GFP) for efficient mask skeleton modeling.<n>Our key innovation is replacing conventional low-level reconstruction with high-level feature prediction that spans from local motion patterns to global semantic representations.
  arXiv  Detail & Related papers  (2025-09-03T18:05:02Z)
• MINR: Implicit Neural Representations with Masked Image Modelling [5.330266804358638]
  Masked autoencoders (MAE) have shown significant promise in learning robust feature representations.<n>We introduce the masked implicit neural representations (MINR) framework that synergizes implicit neural representations with masked image modeling.<n>MINR learns a continuous function to represent images, enabling more robust and generalizable reconstructions irrespective of masking strategies.
  arXiv  Detail & Related papers  (2025-07-30T06:12:57Z)
• Semi-supervised Semantic Segmentation with Multi-Constraint Consistency Learning [81.02648336552421]
  We propose a Multi-Constraint Consistency Learning approach to facilitate the staged enhancement of the encoder and decoder.<n>Self-adaptive feature masking and noise injection are designed in an instance-specific manner to perturb the features for robust learning of the decoder.<n> Experimental results on Pascal VOC2012 and Cityscapes datasets demonstrate that our proposed MCCL achieves new state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-03-23T03:21:33Z)
• RedundancyLens: Revealing and Exploiting Visual Token Processing Redundancy for Efficient Decoder-Only MLLMs [38.34856927170692]
  We propose a training-free framework for analyzing trained Multimodal Large Language Model (MLLM)<n>It consists of Probe-Activated Dynamic FFN and Hollow Attention, which enable adjustable reductions in computations for visual tokens.<n>Experiments demonstrate substantial, structured, and clustered redundancy unique to decoder-only MLLMs.
  arXiv  Detail & Related papers  (2025-01-31T11:09:16Z)
• Cross-Scan Mamba with Masked Training for Robust Spectral Imaging [51.557804095896174]
  We propose the Cross-Scanning Mamba, named CS-Mamba, that employs a Spatial-Spectral SSM for global-local balanced context encoding.<n>Experiment results show that our CS-Mamba achieves state-of-the-art performance and the masked training method can better reconstruct smooth features to improve the visual quality.
  arXiv  Detail & Related papers  (2024-08-01T15:14:10Z)
• Understanding Masked Autoencoders From a Local Contrastive Perspective [80.57196495601826]
  Masked AutoEncoder (MAE) has revolutionized the field of self-supervised learning with its simple yet effective masking and reconstruction strategies. We introduce a new empirical framework, called Local Contrastive MAE, to analyze both reconstructive and contrastive aspects of MAE.
  arXiv  Detail & Related papers  (2023-10-03T12:08:15Z)
• CL-MAE: Curriculum-Learned Masked Autoencoders [49.24994655813455]
  We propose a curriculum learning approach that updates the masking strategy to continually increase the complexity of the self-supervised reconstruction task. We train our Curriculum-Learned Masked Autoencoder (CL-MAE) on ImageNet and show that it exhibits superior representation learning capabilities compared to MAE.
  arXiv  Detail & Related papers  (2023-08-31T09:13:30Z)
• GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds [72.60362979456035]
  Masked Autoencoders (MAE) are challenging to explore in large-scale 3D point clouds. We propose a textbfGenerative textbfDecoder for MAE (GD-MAE) to automatically merges the surrounding context. We demonstrate the efficacy of the proposed method on several large-scale benchmarks: KITTI, and ONCE.
  arXiv  Detail & Related papers  (2022-12-06T14:32:55Z)
• Exploring The Role of Mean Teachers in Self-supervised Masked Auto-Encoders [64.03000385267339]
  Masked image modeling (MIM) has become a popular strategy for self-supervised learning(SSL) of visual representations with Vision Transformers. We present a simple SSL method, the Reconstruction-Consistent Masked Auto-Encoder (RC-MAE) by adding an EMA teacher to MAE. RC-MAE converges faster and requires less memory usage than state-of-the-art self-distillation methods during pre-training.
  arXiv  Detail & Related papers  (2022-10-05T08:08:55Z)
• Rethinking Attention Mechanism in Time Series Classification [6.014777261874646]
  We promote the efficiency and performance of the attention mechanism by proposing our flexible multi-head linear attention (FMLA) We propose a simple but effective mask mechanism that helps reduce the noise influence in time series and decrease the redundancy of the proposed FMLA. We conduct extensive experiments on 85 UCR2018 datasets to compare our algorithm with 11 well-known ones and the results show that our algorithm has comparable performance in terms of top-1 accuracy.
  arXiv  Detail & Related papers  (2022-07-14T07:15:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.