SS3D: End2End Self-Supervised 3D from Web Videos

• URL: http://arxiv.org/abs/2604.22686v1
• Date: Fri, 24 Apr 2026 16:12:44 GMT
• Title: SS3D: End2End Self-Supervised 3D from Web Videos
• Abstract summary: We present SS3D, a web-scale SfM-based self-supervision pretraining pipeline for feed-forward 3D estimation from monocular video.<n>Our model jointly predicts depth, ego-motion, and intrinsics in a single forward pass and is trained/evaluated as a coherent end-to-end 3D estimator.
• Score: 15.931354477455445
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present SS3D, a web-scale SfM-based self-supervision pretraining pipeline for feed-forward 3D estimation from monocular video. Our model jointly predicts depth, ego-motion, and intrinsics in a single forward pass and is trained/evaluated as a coherent end-to-end 3D estimator. To stabilize joint learning, we use an intrinsics-first two-stage schedule and a unified single-checkpoint evaluation protocol. Scaling SfM self-supervision to unconstrained web video is challenging due to weak multi-view observability and strong corpus heterogeneity; we address these with a multi-view signal proxy (MVS) used for filtering and curriculum sampling, and with expert training distilled into a single student. Pretraining on YouTube-8M (~100M frames after filtering) yields strong cross-domain zero-shot transfer and improved fine-tuning performance over prior self-supervised baselines. We release the pretrained checkpoint and code.

Related papers

• Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors [54.51965288351254]
  We present an online 3D vision-language model that enables real-time spatial understanding from streaming video.<n>Our approach adopts an autoregressive streaming control modeling based on the LLM's next-token prediction to learn when to respond.<n>To address the scarcity of streaming 3D-language data, we further develop a scalable data generation pipeline.
  arXiv  Detail & Related papers  (2026-06-05T04:16:24Z)
• Weak to Strong: VLM-Based Pseudo-Labeling as a Weakly Supervised Training Strategy in Multimodal Video-based Hidden Emotion Understanding Tasks [4.888851550406879]
  This paper proposes a weak-vision framework to tackle the automatic recognition of "concealed emotions" in videos.<n>Experiments demonstrate that, despite severe class imbalance, the proposed approach lifts from under 0.6 in prior work to over 0.69.
  arXiv  Detail & Related papers  (2026-02-08T17:02:55Z)
• Learning from Streaming Video with Orthogonal Gradients [62.51504086522027]
  We address the challenge of representation learning from a continuous stream of video as input, in a self-supervised manner.<n>This differs from the standard approaches to video learning where videos are chopped and shuffled during training in order to create a non-redundant batch.<n>We demonstrate the drop in performance when moving from shuffled to sequential learning on three tasks.
  arXiv  Detail & Related papers  (2025-04-02T17:59:57Z)
• 3DMOTFormer: Graph Transformer for Online 3D Multi-Object Tracking [15.330384668966806]
  State-of-the-art 3D multi-object tracking (MOT) approaches typically rely on non-learned model-based algorithms such as Kalman Filter. We propose 3DMOTFormer, a learned geometry-based 3D MOT framework building upon the transformer architecture. Our approach achieves 71.2% and 68.2% AMOTA on the nuScenes validation and test split, respectively.
  arXiv  Detail & Related papers  (2023-08-12T19:19:58Z)
• Learning to Estimate Single-View Volumetric Flow Motions without 3D Supervision [0.0]
  We show that it is possible to train the corresponding networks without requiring any 3D ground truth for training. In the absence of ground truth data we can train our model with observations from real-world capture setups instead of relying on synthetic reconstructions.
  arXiv  Detail & Related papers  (2023-02-28T10:26:02Z)
• 3D-CSL: self-supervised 3D context similarity learning for Near-Duplicate Video Retrieval [17.69904571043164]
  We introduce 3D-SL, a compact pipeline for Near-Duplicate Video Retrieval (NDVR) We propose a two-stage self-supervised similarity learning strategy to optimize the network. Our method achieves the state-of-the-art performance on clip-level NDVR.
  arXiv  Detail & Related papers  (2022-11-10T05:51:08Z)
• SSMTL++: Revisiting Self-Supervised Multi-Task Learning for Video Anomaly Detection [108.57862846523858]
  We revisit the self-supervised multi-task learning framework, proposing several updates to the original method. We modernize the 3D convolutional backbone by introducing multi-head self-attention modules. In our attempt to further improve the model, we study additional self-supervised learning tasks, such as predicting segmentation maps.
  arXiv  Detail & Related papers  (2022-07-16T19:25:41Z)
• Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training [56.81809311892475]
  Masked Autoencoders (MAE) have shown great potentials in self-supervised pre-training for language and 2D image transformers. We propose Point-M2AE, a strong Multi-scale MAE pre-training framework for hierarchical self-supervised learning of 3D point clouds.
  arXiv  Detail & Related papers  (2022-05-28T11:22:53Z)
• PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision [102.48681650013698]
  Existing self-supervised 3D human pose estimation schemes have largely relied on weak supervisions to guide the learning. We propose a novel self-supervised approach that allows us to explicitly generate 2D-3D pose pairs for augmenting supervision. This is made possible via introducing a reinforcement-learning-based imitator, which is learned jointly with a pose estimator alongside a pose hallucinator.
  arXiv  Detail & Related papers  (2022-03-29T14:45:53Z)
• Long-Short Temporal Contrastive Learning of Video Transformers [62.71874976426988]
  Self-supervised pretraining of video transformers on video-only datasets can lead to action recognition results on par or better than those obtained with supervised pretraining on large-scale image datasets. Our approach, named Long-Short Temporal Contrastive Learning, enables video transformers to learn an effective clip-level representation by predicting temporal context captured from a longer temporal extent.
  arXiv  Detail & Related papers  (2021-06-17T02:30:26Z)
• SeCo: Exploring Sequence Supervision for Unsupervised Representation Learning [114.58986229852489]
  In this paper, we explore the basic and generic supervision in the sequence from spatial, sequential and temporal perspectives. We derive a particular form named Contrastive Learning (SeCo) SeCo shows superior results under the linear protocol on action recognition, untrimmed activity recognition and object tracking.
  arXiv  Detail & Related papers  (2020-08-03T15:51:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.