Towards Stable Self-Supervised Object Representations in Unconstrained Egocentric Video

• URL: http://arxiv.org/abs/2603.13912v1
• Date: Sat, 14 Mar 2026 12:00:55 GMT
• Title: Towards Stable Self-Supervised Object Representations in Unconstrained Egocentric Video
• Abstract summary: We propose a unified vision Transformer framework designed to learn stable object representations from unlabeled egocentric video.<n>EgoViT bootstraps this learning process by jointly discovering and stabilizing "proto-objects"<n>EgoViT achieves +8.0% CorLoc improvement in unsupervised object discovery and +4.8% mIoU improvement in semantic segmentation.
• Score: 8.642846048553041
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Humans develop visual intelligence through perceiving and interacting with their environment - a self-supervised learning process grounded in egocentric experience. Inspired by this, we ask how can artificial systems learn stable object representations from continuous, uncurated first-person videos without relying on manual annotations. This setting poses challenges of separating, recognizing, and persistently tracking objects amid clutter, occlusion, and ego-motion. We propose EgoViT, a unified vision Transformer framework designed to learn stable object representations from unlabeled egocentric video. EgoViT bootstraps this learning process by jointly discovering and stabilizing "proto-objects" through three synergistic mechanisms: (1) Proto-object Learning, which uses intra-frame distillation to form discriminative representations; (2) Depth Regularization, which grounds these representations in geometric structure; and (3) Teacher-Filtered Temporal Consistency, which enforces identity over time. This creates a virtuous cycle where initial object hypotheses are progressively refined into stable, persistent representations. The framework is trained end-to-end on unlabeled first-person videos and exhibits robustness to geometric priors of varied origin and quality. On standard benchmarks, EgoViT achieves +8.0% CorLoc improvement in unsupervised object discovery and +4.8% mIoU improvement in semantic segmentation, demonstrating its potential to lay a foundation for robust visual abstraction in embodied intelligence.

Related papers

• EgoMotion: Hierarchical Reasoning and Diffusion for Egocentric Vision-Language Motion Generation [74.07852250099559]
  We investigate Egocentric Vision-Language (Ego-VL) motion generation.<n>This task requires synthesizing 3D human motion conditioned jointly on first-person visual observations and natural language instructions.<n>We propose a hierarchical generative framework textbfEgoMotion to address this challenge.
  arXiv  Detail & Related papers  (2026-04-21T05:31:06Z)
• Ego-InBetween: Generating Object State Transitions in Ego-Centric Videos [56.20829168540647]
  EgoIn is a framework that infers the multi-step transition process between two given states using TransitionVLM.<n>It generates a sequence of frames based on transition conditions produced by the proposed Transition Conditioning module.<n>Experiments on human-object and robot-object interaction datasets demonstrate EgoIn's superior performance in generating semantically meaningful and visually coherent transformation sequences.
  arXiv  Detail & Related papers  (2026-04-20T03:07:22Z)
• EgoForge: Goal-Directed Egocentric World Simulator [26.712565464146937]
  EgoForge is a goal-directed world simulator that generates coherent, first-person video rollouts from minimal static inputs.<n>VideoDiffusionNFT is a trajectory-level reward-guided refinement that optimize goal completion, temporal causality, scene consistency, and perceptual fidelity during diffusion sampling.
  arXiv  Detail & Related papers  (2026-03-20T17:46:55Z)
• Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention [58.05340906967343]
  Egocentric Referring Video Object (Ego-RVOS) aims to segment the specific object actively involved in a human action, as described by a language query, within first-person videos.<n>Existing methods often struggle, learning spurious correlations from skewed object-action pairings in datasets.<n>We introduce Causal-REferring (CERES), a plug-in causal framework that adapts strong, pre-trained RVOSs to the egocentric domain.
  arXiv  Detail & Related papers  (2025-12-30T16:22:14Z)
• Temporally Consistent Object-Centric Learning by Contrasting Slots [23.203973564679508]
  We introduce a novel object-level temporal contrastive loss for video object-centric models.<n>Our method significantly improves the temporal consistency of the learned object-centric representations.
  arXiv  Detail & Related papers  (2024-12-18T19:46:04Z)
• Object-Centric Temporal Consistency via Conditional Autoregressive Inductive Biases [69.46487306858789]
  Conditional Autoregressive Slot Attention (CA-SA) is a framework that enhances the temporal consistency of extracted object-centric representations in video-centric vision tasks. We present qualitative and quantitative results showing that our proposed method outperforms the considered baselines on downstream tasks.
  arXiv  Detail & Related papers  (2024-10-21T07:44:44Z)
• Learning Object Permanence from Videos via Latent Imaginations [4.293591000527724]
  We introduce a slot-based autoregressive deep learning system, the looped location and identity tracking model Loci-Looped. We show that Loci-Looped learns to adaptively fuse latent imaginations with pixel-space observations into consistent latent object-specific what and where encodings over time. Our work thus introduces the first self-supervised interpretable learning model that learns about object permanence directly from video data without supervision.
  arXiv  Detail & Related papers  (2023-10-16T13:11:35Z)
• Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignment [71.16699226211504]
  We propose to learn fine-grained action features that are invariant to the viewpoints by aligning egocentric and exocentric videos in time. To this end, we propose AE2, a self-supervised embedding approach with two key designs. For evaluation, we establish a benchmark for fine-grained video understanding in the ego-exo context.
  arXiv  Detail & Related papers  (2023-06-08T19:54:08Z)
• Stochastic Coherence Over Attention Trajectory For Continuous Learning In Video Streams [64.82800502603138]
  This paper proposes a novel neural-network-based approach to progressively and autonomously develop pixel-wise representations in a video stream. The proposed method is based on a human-like attention mechanism that allows the agent to learn by observing what is moving in the attended locations. Our experiments leverage 3D virtual environments and they show that the proposed agents can learn to distinguish objects just by observing the video stream.
  arXiv  Detail & Related papers  (2022-04-26T09:52:31Z)
• Self-Regulated Learning for Egocentric Video Activity Anticipation [147.9783215348252]
  Self-Regulated Learning (SRL) aims to regulate the intermediate representation consecutively to produce representation that emphasizes the novel information in the frame of the current time-stamp. SRL sharply outperforms existing state-of-the-art in most cases on two egocentric video datasets and two third-person video datasets.
  arXiv  Detail & Related papers  (2021-11-23T03:29:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.