FROST-STA: Frozen Dense Features for the Ego4D Short-Term Object Interaction Anticipation

• URL: http://arxiv.org/abs/2606.00694v1
• Date: Sat, 30 May 2026 12:07:32 GMT
• Title: FROST-STA: Frozen Dense Features for the Ego4D Short-Term Object Interaction Anticipation
• Abstract summary: This report describes FROST-STA, our submission to the Ego4D Short-Term Object Interaction Anticipation (STA) Challenge at EgoVis 2026.<n>For each query time, the model produces a ranked set of structured hypotheses containing an active-object box, noun label, verb label, time-to-contact (TTC), and confidence.<n> FROST-STA builds on the V-JEPA 2.1 STA evaluation protocol, but adapts it to the challenge by using object-centric decoding, multi-head prediction, and a submission-oriented training and ensembling recipe.
• Score: 4.781417767575192
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Short-term anticipation in egocentric video requires more than recognizing the current scene: a system must infer which object the camera wearer will contact, which action will follow, and how soon the contact will happen. This report describes FROST-STA, our submission to the Ego4D Short-Term Object Interaction Anticipation (STA) Challenge at EgoVis 2026. For each query time, the model produces a ranked set of structured hypotheses containing an active-object box, noun label, verb label, time-to-contact (TTC), and confidence. FROST-STA builds on the V-JEPA 2.1 STA evaluation protocol, but adapts it to the challenge by using object-centric decoding, multi-head prediction, and a submission-oriented training and ensembling recipe. We keep the V-JEPA 2.1 ViT-G backbone fixed and extract two dense token streams: video tokens from a short clip resized to 384 pixels before the query, and image tokens from the last observed high-resolution frame. A compact alignment module, consisting of an attentive probe and frame-guided temporal pooling, maps the clip representation onto the spatial reference of the final frame before fusing it with image features. The fused maps are decoded by Faster R-CNN-style STA heads that estimate box offsets, nouns, verbs, TTC values, and interaction quality. For the final leaderboard entry, we train for 25 epochs with the official training split plus additional permitted validation annotations, and combine predictions across eight heads and checkpoints from epochs 15-25. FROST-STA obtains 5.13 Overall Top-5 mAP on the official test server, ranking second in the challenge and showing that frozen dense image-video features can serve as a strong basis for object-level interaction forecasting.

Related papers

• Predict, Then Retrieve: Cross-Instance Future-State Retrieval from Video Prefixes [39.17422574355306]
  We introduce Predictive State Retrieval (PSR), a task in which a model observes a short video prefix and a temporal question about an object's future state.<n>PSR combines anticipation with cross-instance retrieval across multiple temporal horizons.<n>We also propose LFTR, a lightweight retriever with frozen encoders that predicts a question- and horizon-conditioned future latent.
  arXiv  Detail & Related papers  (2026-08-05T04:12:00Z)
• EgoAction: Egocentric Action Composition with Reliability-Aware Temporal Fusion for the EPIC-KITCHENS Action Detection Challenge at CVPR 2026 [69.56534058291463]
  We formulate our submission as EgoAction, a unified decoupled detection and fusion pipeline.<n>The pipeline uses EPIC-finetuned VideoMAE-L features, trains separate noun and verb temporal detectors with causal temporal modeling.<n>EgoAction provides a compact and reproducible system for egocentric temporal action detection.
  arXiv  Detail & Related papers  (2026-05-23T10:05:56Z)
• TempRet: Temporal Enhancement and Two-Stage Reranking for CVPR 2026 EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge [71.10535279591527]
  We present our solution, termed TempRet, to the CVPR 2026 EPIC-KITCHENS-100 MIR challenge.<n>Our approach builds upon a CLIP-based dual-encoder backbone and introduces two key components to address the temporal and cross-modal challenges.<n>Our method achieves 67.97% average mAP and 82.92% average nDCG on the EK-100 MIR benchmark.
  arXiv  Detail & Related papers  (2026-05-23T08:37:39Z)
• VISTA: Technical Report for the Ego4D Short-Term Object Interaction Anticipation at EgoVis 2026 [70.67609435035631]
  Given an egocentric video timestamp, the task requires anticipating the next human-object interaction.<n>ViSTA follows a StillFast-style design that combines object-centric spatial detection with short-horizon temporal context.<n>Experiments show that VISTA achieves first place in the EgoVis 2026 Ego4D STA Challenge.
  arXiv  Detail & Related papers  (2026-05-20T08:42:56Z)
• PEARL: Personalized Streaming Video Understanding Model [50.273809779498464]
  multimodal personalization methods are largely limited to static images or offline videos.<n>PEARL-Bench is the first comprehensive benchmark designed specifically to evaluate this challenging setting.<n>PEARL achieves state-of-the-art performance across 8 offline and online models.
  arXiv  Detail & Related papers  (2026-03-20T18:47:54Z)
• Integrating Affordances and Attention models for Short-Term Object Interaction Anticipation [17.55088626523059]
  Short Term object-interaction Anticipation is fundamental for wearable assistants to understand user goals and provide timely assistance.<n>In this work, we present a method to improve the performance of STA predictions.<n>Our results show significant improvements on Overall Top-5 mAP, with gain up to +23p.p on Ego4D and +31p.p on a novel set of curated EPIC-Kitchens STA labels.
  arXiv  Detail & Related papers  (2026-02-16T15:29:04Z)
• FRAME: Pre-Training Video Feature Representations via Anticipation and Memory [55.046881477209695]
  FRAME is a self-supervised video frame encoder tailored for dense video understanding.<n>It learns to predict current and future DINO patch features from past and present RGB frames.<n>It consistently outperforms image encoders and existing self-supervised video models.
  arXiv  Detail & Related papers  (2025-06-05T19:44:47Z)
• ZARRIO @ Ego4D Short Term Object Interaction Anticipation Challenge: Leveraging Affordances and Attention-based models for STA [10.144283429670807]
  Short-Term object-interaction Anticipation (STA) consists of detecting the location of the next-active objects, the noun and verb categories of the interaction, and the time to contact from the observation of egocentric video. We propose STAformer, a novel attention-based architecture integrating frame-guided temporal pooling, dual image-video attention, and multi-scale feature fusion to support STA predictions from an image-input video pair.
  arXiv  Detail & Related papers  (2024-07-05T09:16:30Z)
• AFF-ttention! Affordances and Attention models for Short-Term Object Interaction Anticipation [14.734158936250918]
  Short-Term object-interaction Anticipation is fundamental for wearable assistants or human robot interaction to understand user goals. We improve the performance of STA predictions with two contributions. First, we propose STAformer, a novel attention-based architecture integrating frame guided temporal pooling, dual image-video attention, and multiscale feature fusion. Second, we predict interaction hotspots from the observation of hands and object trajectories, increasing confidence in STA predictions localized around the hotspot.
  arXiv  Detail & Related papers  (2024-06-03T10:57:18Z)
• Structured Video Tokens @ Ego4D PNR Temporal Localization Challenge 2022 [93.98605636451806]
  This report describes the SViT approach for the Ego4D Point of No Return (PNR) Temporal Localization Challenge. We propose a learning framework which demonstrates how utilizing the structure of a small number of images only available during training can improve a video model. SViT obtains strong performance on the challenge test set with 0.656 absolute temporal localization error.
  arXiv  Detail & Related papers  (2022-06-15T17:36:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.