One for All: Generalist Foundation Model for Cross-Sensor Skeleton Representation Learning

• URL: http://arxiv.org/abs/2609.07078v1
• Date: Mon, 07 Sep 2026 06:05:44 GMT
• Title: One for All: Generalist Foundation Model for Cross-Sensor Skeleton Representation Learning
• Abstract summary: We introduce SOfA (Skeleton One for All) to achieve sensor-unified skeleton representation learning across diverse sensors.<n>To accommodate the dimensional gap caused by varying joint counts, we introduce a fixed-size set of learnable Canonical Joint Slots.<n>SOfA fills these slots via an attention mechanism that dynamically aggregates skeletal information from sensor-specific inputs.
• Score: 30.16840983345016
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: For learning generalizable motion representations from large-scale unlabeled data, Self-supervised learning (SSL) has become a widely adopted methodology. However, existing approaches are primarily limited by the inherent heterogeneity of skeleton data---characterized by varying joint counts, indexing protocols, and topological structures across different sensors---which typically necessitates training separate, sensor-specific, or even entirely dataset-specific models. To overcome this, we introduce SOfA (Skeleton One for All), the first generalist foundation model designed to achieve sensor-unified skeleton representation learning across diverse sensors. To accommodate the dimensional gap caused by varying joint counts, we introduce a fixed-size set of learnable Canonical Joint Slots, acting as a universal vessel that seamlessly accommodates arbitrary skeletal topologies. SOfA fills these slots via an attention mechanism that dynamically aggregates skeletal information from sensor-specific inputs. Furthermore, we resolve joint index misalignment between various sensors by introducing a Semantic Joint Embedding derived from a pre-trained text encoder, rather than relying on absolute positional embeddings. To validate our approach, we standardized ten 3D skeleton datasets for unified training. Extensive experiments demonstrate that SOfA can serve as a truly universal encoder, achieving state-of-the-art (SOTA) performance across a wide range of downstream tasks and sensor types, often outperforming dataset-specific specialist models with a single foundation model.

Related papers

• SSC-Priors: Exploring Semantic and Visibility Priors to Boost Lidar Semantic Scene Completion [61.665449818739155]
  This paper investigates easy strategies to boost the performance of existing networks for lidar semantic scene completion (SSC)<n>We use semantic pseudo-labels and sensor visibility information to improve existing approaches.<n>We show that endowing input point clouds with semantic pseudo-labels from off-the-shelf segmenters significantly improves the performance of existing SSC models.
  arXiv  Detail & Related papers  (2026-09-15T16:37:46Z)
• One Model to Magnify Them All: Efficient Scale-Invariant Histopathology via Conditional Normalization and Continuous Magnification Training [2.519619892651923]
  Deep learning models are sensitive to scale variation.<n>We propose Conditional Normalization (CLN), a mechanism that generates affine parameters from pixel size via small pixel sizes.<n>On the PANDA prostate cancer dataset, our approach matches or independently trained single-magnification models and ranks among the top three performers at every evaluated magnification.
  arXiv  Detail & Related papers  (2026-08-10T10:28:43Z)
• Skeleton-to-Image Encoding: Enabling Skeleton Representation Learning via Vision-Pretrained Models [110.11712022072975]
  We introduce Skeleton-to-Image, a novel representation that transforms skeleton sequences into image-like data.<n>This encoding enables the use of powerful vision-pretrained models for self-supervised skeleton representation learning.
  arXiv  Detail & Related papers  (2026-03-06T06:54:55Z)
• One Language-Free Foundation Model Is Enough for Universal Vision Anomaly Detection [65.11602552904456]
  Universal visual anomaly detection (AD) aims to identify anomaly images and segment anomaly regions towards open and dynamic scenarios.<n>Current methods often struggle with complex prompt engineering, elaborate adaptation modules, and challenging training strategies.<n>This paper presents an embarrassingly simple, general, and effective framework for Universal vision Anomaly Detection (UniADet)
  arXiv  Detail & Related papers  (2026-01-09T06:05:18Z)
• SpecAware: A Spectral-Content Aware Foundation Model for Unifying Multi-Sensor Learning in Hyperspectral Remote Sensing Mapping [9.392313789126714]
  SpecAware is a novel hyperspectral spectral-content aware foundation model for unifying multi-sensor learning for HSI mapping.<n>The core of SpecAware is a two-step hypernetwork-driven encoding process for HSI data.<n>Experiments on six datasets demonstrate that SpecAware can learn superior feature representations.
  arXiv  Detail & Related papers  (2025-10-31T06:28:14Z)
• CoPS: Conditional Prompt Synthesis for Zero-Shot Anomaly Detection [6.1568149026052374]
  Conditional Prompt Synthesis (CoPS) is a novel framework that synthesizes dynamic prompts conditioned on visual features to enhance ZSAD performance.<n>CoPS surpasses state-of-the-art methods by 2.5% AUROC in both classification and segmentation across 13 industrial and medical datasets.
  arXiv  Detail & Related papers  (2025-08-05T13:47:45Z)
• Private Training & Data Generation by Clustering Embeddings [74.00687214400021]
  Differential privacy (DP) provides a robust framework for protecting individual data.<n>We introduce a novel principled method for DP synthetic image embedding generation.<n> Empirically, a simple two-layer neural network trained on synthetically generated embeddings achieves state-of-the-art (SOTA) classification accuracy.
  arXiv  Detail & Related papers  (2025-06-20T00:17:14Z)
• Spatial-Temporal-Spectral Unified Modeling for Remote Sensing Dense Prediction [20.1863553357121]
  Current deep learning architectures for remote sensing are fundamentally rigid.<n>We introduce the Spatial-Temporal-Spectral Unified Network (STSUN) for unified modeling.<n> STSUN can adapt to input and output data with arbitrary spatial sizes, temporal lengths, and spectral bands.<n>It unifies various dense prediction tasks and diverse semantic class predictions.
  arXiv  Detail & Related papers  (2025-05-18T07:39:17Z)
• Toward Foundation Model for Multivariate Wearable Sensing of Physiological Signals [2.370585289844609]
  We propose the first multi-modal and ubiquitous foundation model designed to extract informative representations from wearable sensing data.<n>Specifically, we design a channel-aware attention mechanism with a shared special liaison token to detect signal patterns in both intra-sensor and inter-sensors.<n>Our model shows exceptional generalizability across 11 public wearable sensing datasets, spanning 18 applications in mental health, body state inference, vital sign estimation, and disease risk evaluation.
  arXiv  Detail & Related papers  (2024-12-12T23:35:18Z)
• A Generically Contrastive Spatiotemporal Representation Enhancement for 3D Skeleton Action Recognition [10.403751563214113]
  We propose a Contrastive Spatiotemporal Representation Enhancement (CSRE) framework to obtain more discriminative representations from the sequences.<n>Specifically, we decompose the representation into spatial- and temporal-specific features to explore fine-grained motion patterns.<n>To explicitly exploit the latent data distributions, we employ the attentive features to contrastive learning, which models the cross-sequence semantic relations.
  arXiv  Detail & Related papers  (2023-12-23T02:54:41Z)
• Domain Adaptive Synapse Detection with Weak Point Annotations [63.97144211520869]
  We present AdaSyn, a framework for domain adaptive synapse detection with weak point annotations. In the WASPSYN challenge at I SBI 2023, our method ranks the 1st place.
  arXiv  Detail & Related papers  (2023-08-31T05:05:53Z)
• SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-training [110.55093254677638]
  We propose an efficient skeleton sequence learning framework, named Skeleton Sequence Learning (SSL) In this paper, we build an asymmetric graph-based encoder-decoder pre-training architecture named SkeletonMAE. Our SSL generalizes well across different datasets and outperforms the state-of-the-art self-supervised skeleton-based action recognition methods.
  arXiv  Detail & Related papers  (2023-07-17T13:33:11Z)
• FSAR: Federated Skeleton-based Action Recognition with Adaptive Topology Structure and Knowledge Distillation [23.0771949978506]
  Existing skeleton-based action recognition methods typically follow a centralized learning paradigm, which can pose privacy concerns when exposing human-related videos. We introduce a novel Federated Skeleton-based Action Recognition (FSAR) paradigm, which enables the construction of a globally generalized model without accessing local sensitive data.
  arXiv  Detail & Related papers  (2023-06-19T16:18:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.