UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning

• URL: http://arxiv.org/abs/2606.20559v1
• Date: Thu, 18 Jun 2026 17:59:45 GMT
• Title: UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning
• Abstract summary: UNIEGO is a unified egocentric encoder trained with nine teachers spanning ego-exo viewpoints, RGB, depth, and skeleton modalities, and four foundation models.<n>We demonstrate that structured, proxy-mediated knowledge transfer yields richer and more discriminative egocentric representations.
• Score: 20.624929008039402
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Egocentric video understanding is inherently limited by the narrow perspective of wearable cameras: a single viewpoint, a single modality, a single model cannot capture the full richness of human action. We argue that a truly expressive egocentric representation must subsume complementary knowledge across viewpoints, modalities, and foundation model representations, yet remain deployable from egocentric video alone. To this end, we introduce a hierarchical multi-teacher distillation framework that produces UNIEGO, a unified egocentric encoder trained with nine teachers spanning ego-exo viewpoints, RGB, depth, and skeleton modalities, and four foundation models. Rather than distilling directly from heterogeneous teachers whose incompatible architectures and feature geometries induce conflicting gradients, our framework interposes a layer of representation-specific Proxy models that translate diverse teacher knowledge into a homogeneous egocentric space. A second distillation stage, Selective Proxy Distillation (SPD), then adaptively selects, for each training sample, the subset of proxies that are both correct and confident, distilling exclusively from reliable supervision and suppressing erroneous signals. SPD is further stabilized by initializing UNIEGO as a learned convex combination of proxy parameters, placing the unified model in a well-conditioned region of the loss landscape before distillation begins. UNIEGO achieves state-of-the-art performance across three egocentric video understanding tasks - action recognition, video retrieval, and action segmentation on three challenging ego-exo benchmarks, outperforming naive multi-teacher distillation baselines and demonstrating that structured, proxy-mediated knowledge transfer yields richer and more discriminative egocentric representations.

Related papers

• Heterogeneous and Adept Snapshot Distillation for 3D Semantic Segmentation [71.94615164565589]
  Multi-modal fusion and multi-model ensembling are prevalent in enhancing the performance of 3D semantic segmentation.<n>We propose to transfer the rich knowledge from the multi-modal model (i.e., point clouds and images) and multiple model experts to the point-cloud-based network through knowledge distillation.<n>The resulting Heterogeneous and Adept Snapshot Knowledge Distillation, dubbed HAS-KD, attains state-of-the-art results on ScanNetV2 and S3DIS datasets.
  arXiv  Detail & Related papers  (2026-06-24T01:31:22Z)
• Divide, Deliberate, Decide: A Multi-Agent Framework for Fine-Grained Egocentric Action Recognition [86.4487145812318]
  Fine-grained action recognition in egocentric video is challenging for Vision-Language Models.<n>We propose Divide, Deliberate, Decide, a fully-local, zero-shot multi-agent framework.<n> Experiments show that our method positively improves zero-shot action recognition performance over the baseline.
  arXiv  Detail & Related papers  (2026-06-16T07:31:27Z)
• Towards Stable Self-Supervised Object Representations in Unconstrained Egocentric Video [8.642846048553041]
  We propose a unified vision Transformer framework designed to learn stable object representations from unlabeled egocentric video.<n>EgoViT bootstraps this learning process by jointly discovering and stabilizing "proto-objects"<n>EgoViT achieves +8.0% CorLoc improvement in unsupervised object discovery and +4.8% mIoU improvement in semantic segmentation.
  arXiv  Detail & Related papers  (2026-03-14T12:00:55Z)
• Brewing Stronger Features: Dual-Teacher Distillation for Multispectral Earth Observation [0.0]
  Foundation models are transforming Earth Observation (EO), yet the diversity of EO sensors and modalities makes a single universal model unrealistic.<n>We propose a dual-teacher contrastive distillation framework for multispectral imagery.<n>Our approach combines a multispectral teacher with an optical VFM teacher, enabling coherent cross-modal representation learning.
  arXiv  Detail & Related papers  (2026-02-23T14:09:01Z)
• Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention [58.05340906967343]
  Egocentric Referring Video Object (Ego-RVOS) aims to segment the specific object actively involved in a human action, as described by a language query, within first-person videos.<n>Existing methods often struggle, learning spurious correlations from skewed object-action pairings in datasets.<n>We introduce Causal-REferring (CERES), a plug-in causal framework that adapts strong, pre-trained RVOSs to the egocentric domain.
  arXiv  Detail & Related papers  (2025-12-30T16:22:14Z)
• DDAE++: Enhancing Diffusion Models Towards Unified Generative and Discriminative Learning [53.27049077100897]
  generative pre-training has been shown to yield discriminative representations, paving the way towards unified visual generation and understanding.<n>This work introduces self-conditioning, a mechanism that internally leverages the rich semantics inherent in denoising network to guide its own decoding layers.<n>Results are compelling: our method boosts both generation FID and recognition accuracy with 1% computational overhead and generalizes across diverse diffusion architectures.
  arXiv  Detail & Related papers  (2025-05-16T08:47:16Z)
• Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understanding [69.96199605596138]
  Current MLLMs primarily focus on third-person (exocentric) vision, overlooking the unique aspects of first-person (egocentric) videos.<n>We propose learning the mapping between exocentric and egocentric domains to enhance egocentric video understanding.<n>We introduce Ego-ExoClip, a pre-training dataset comprising 1.1M synchronized ego-exo clip-text pairs.
  arXiv  Detail & Related papers  (2025-03-12T08:10:33Z)
• Semi-supervised learning made simple with self-supervised clustering [65.98152950607707]
  Self-supervised learning models have been shown to learn rich visual representations without requiring human annotations. We propose a conceptually simple yet empirically powerful approach to turn clustering-based self-supervised methods into semi-supervised learners.
  arXiv  Detail & Related papers  (2023-06-13T01:09:18Z)
• Knowledge Distillation Meets Self-Supervision [109.6400639148393]
  Knowledge distillation involves extracting "dark knowledge" from a teacher network to guide the learning of a student network. We show that the seemingly different self-supervision task can serve as a simple yet powerful solution. By exploiting the similarity between those self-supervision signals as an auxiliary task, one can effectively transfer the hidden information from the teacher to the student.
  arXiv  Detail & Related papers  (2020-06-12T12:18:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.