Joint-Embedding Prediction of Masked Point Tubes for Self-Supervised Learning on 4D Point Cloud Videos

• URL: http://arxiv.org/abs/2608.24093v1
• Date: Tue, 25 Aug 2026 05:38:43 GMT
• Title: Joint-Embedding Prediction of Masked Point Tubes for Self-Supervised Learning on 4D Point Cloud Videos
• Abstract summary: We propose a JEPA-style framework that learns from unsupervised point clouds through latent point-temporal prediction.<n>We show that the learned representations improve downstream fine-level annotations, limited-level annotations, and cross-dataset transfer.<n>These results suggest that JEPA-style prediction is a promising alternative to reconstruction-centered pretraining for 4D point cloud videos.
• Score: 8.592776018057192
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised representation learning for 4D point cloud videos is challenging because annotations are costly and reconstruction-based pretraining can overemphasize low-level geometric details. We propose a JEPA-style framework that learns from unlabeled spatiotemporal point clouds through latent point-tube prediction. Instead of reconstructing raw coordinates, the model masks spatiotemporal regions and predicts their target representations from visible context representations in feature space. To stabilize latent prediction, we incorporate Sketched Isotropic Gaussian Regularization, which encourages non-collapsed embeddings without relying on explicit reconstruction targets. This formulation aims to capture both spatial structure and temporal dynamics while keeping the pretraining objective aligned with downstream semantic recognition. Experiments on action and gesture recognition benchmarks show that the learned representations improve downstream fine-tuning, limited-label learning, and cross-dataset transfer. These results suggest that JEPA-style latent prediction is a promising alternative to reconstruction-centered pretraining for 4D point cloud videos.

Related papers

• DynaTok: Token-Based 4D Reconstruction from Partial Point Clouds [86.5916611076068]
  DynaTok is a point-based framework for correspondence-free 4D reconstruction from partial point sequences without images.<n>A flow-matching decoder then reconstructs complete, temporally consistent 4D point-temporal observations.
  arXiv  Detail & Related papers  (2026-06-10T15:13:14Z)
• Momentum-Guided Semantic Forecasting (MoFore) for Self-Supervised Video Representation Learning [0.0]
  We introduce a Momentum-Guided Semantic Forecasting framework for self-supervised video representation learning.<n>The framework learns temporally consistent and semantically meaningful video representations without using action labels during training.
  arXiv  Detail & Related papers  (2026-06-08T17:50:01Z)
• Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds [53.82500407523346]
  PointINS is an instance-oriented self-supervised framework that enriches point cloud representations through geometry-aware learning.<n>PointINS achieves on average +3.5% mAP improvement for indoor instance segmentation and +4.1% PQ gain for outdoor panoptic segmentation.
  arXiv  Detail & Related papers  (2026-03-26T08:31:06Z)
• Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking [52.393359791978035]
  Motion2VecSets is a 4D diffusion model for dynamic surface reconstruction from point cloud sequences. We parameterize 4D dynamics with latent sets instead of using global latent codes. For more temporally-coherent object tracking, we synchronously denoise deformation latent sets and exchange information across multiple frames.
  arXiv  Detail & Related papers  (2024-01-12T15:05:08Z)
• Point Contrastive Prediction with Semantic Clustering for Self-Supervised Learning on Point Cloud Videos [71.20376514273367]
  We propose a unified point cloud video self-supervised learning framework for object-centric and scene-centric data. Our method outperforms supervised counterparts on a wide range of downstream tasks.
  arXiv  Detail & Related papers  (2023-08-18T02:17:47Z)
• Masked Spatio-Temporal Structure Prediction for Self-supervised Learning on Point Cloud Videos [75.9251839023226]
  We propose a Masked-temporal Structure Prediction (MaST-Pre) method to capture the structure of point cloud videos without human annotations. MaST-Pre consists of two self-supervised learning tasks. First, by reconstructing masked point tubes, our method is able to capture appearance information of point cloud videos. Second, to learn motion, we propose a temporal cardinality difference prediction task that estimates the change in the number of points within a point tube.
  arXiv  Detail & Related papers  (2023-08-18T02:12:54Z)
• Contrastive Predictive Autoencoders for Dynamic Point Cloud Self-Supervised Learning [26.773995001469505]
  We design point cloud sequence based Contrastive Prediction and Reconstruction (CPR), to collaboratively learn more comprehensive representations. We conduct experiments on four point cloud sequence benchmarks, and report the results under multiple experimental settings.
  arXiv  Detail & Related papers  (2023-05-22T12:09:51Z)
• GeoMAE: Masked Geometric Target Prediction for Self-supervised Point Cloud Pre-Training [16.825524577372473]
  We introduce a point cloud representation learning framework, based on geometric feature reconstruction. We identify three self-supervised learning objectives to peculiar point clouds, namely centroid prediction, normal estimation, and curvature prediction. Our pipeline is conceptually simple and it consists of two major steps: first, it randomly masks out groups of points, followed by a Transformer-based point cloud encoder.
  arXiv  Detail & Related papers  (2023-05-15T17:14:55Z)
• RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction [19.535169371240073]
  We introduce RfD-Net that jointly detects and reconstructs dense object surfaces directly from point clouds. We decouple the instance reconstruction into global object localization and local shape prediction. Our approach consistently outperforms the state-of-the-arts and improves over 11 of mesh IoU in object reconstruction.
  arXiv  Detail & Related papers  (2020-11-30T12:58:05Z)
• CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations [72.4716073597902]
  We propose a method to learn object Canonical Point Cloud Representations of dynamically or moving objects. We demonstrate the effectiveness of our method on several applications including shape reconstruction, camera pose estimation, continuoustemporal sequence reconstruction, and correspondence estimation.
  arXiv  Detail & Related papers  (2020-08-06T17:58:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.