TAP-JEPA: Frozen Future-Latent Probing and Two-Stage Score Fusion for EPIC-KITCHENS-100 Action Anticipation

• URL: http://arxiv.org/abs/2606.00662v1
• Date: Sat, 30 May 2026 10:23:12 GMT
• Title: TAP-JEPA: Frozen Future-Latent Probing and Two-Stage Score Fusion for EPIC-KITCHENS-100 Action Anticipation
• Abstract summary: This report presents TAP-JEPA, our runner-up submission to the EPIC-KITCHENS-100 (EK-100) Action Anticipation Challenge at EgoVis 2026.<n>The task is to anticipate the next verb, noun, and verb-noun action from an egocentric clip that ends before the target action begins.<n>Instead of fine-tuning a large video backbone, TAP-JEPA builds a compact anticipation model on frozen V-JEPA 2.1 features.
• Score: 4.781417767575192
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This report presents TAP-JEPA, our runner-up submission to the EPIC-KITCHENS-100 (EK-100) Action Anticipation Challenge at EgoVis 2026. The task is to anticipate the next verb, noun, and verb-noun action from an egocentric clip that ends before the target action begins. Instead of fine-tuning a large video backbone, TAP-JEPA builds a compact anticipation model on frozen V-JEPA 2.1 features: a ViT-G/384 encoder extracts visible pre-action tokens, the pre-trained latent predictor estimates near-future tokens from the observed context, and both token groups are fused by attentive probes with task-specific queries for verbs, nouns, and action pairs. For the final submission, we expand supervised training with the official training split and most of the validation split, reserving a small subset for sanity checks and qualitative inspection, and adopt a two-stage score fusion that first averages eight independently initialized probe replicas within each epoch and then merges candidates from epochs 12-20 with field-dependent weights. On the official open-testing leaderboard, our sunshinesky entry achieves 27.91 percent overall action Mean Top-5 Recall (MT5R), ranking second and only 0.04 percentage points behind the top score.

Related papers

• GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation [63.853929983870955]
  We introduce Genie Envisioner Act 2.0 (GE-Act 2.0), a world-action model whose trainable generative and action components are all from scratch on manipulation data.<n>It combines a control-oriented autoencoder (CoAE), a single-step visual planner (SVP), and an inverse dynamics model (IDM)<n>The components are then jointly trained with knowledge-aligned selective optimization (KASO), which reduces mismatched supervision by selecting only predicted futures judged behaviorally compatible with the recorded action.
  arXiv  Detail & Related papers  (2026-09-04T17:16:48Z)
• FROST-STA: Frozen Dense Features for the Ego4D Short-Term Object Interaction Anticipation [4.781417767575192]
  This report describes FROST-STA, our submission to the Ego4D Short-Term Object Interaction Anticipation (STA) Challenge at EgoVis 2026.<n>For each query time, the model produces a ranked set of structured hypotheses containing an active-object box, noun label, verb label, time-to-contact (TTC), and confidence.<n> FROST-STA builds on the V-JEPA 2.1 STA evaluation protocol, but adapts it to the challenge by using object-centric decoding, multi-head prediction, and a submission-oriented training and ensembling recipe.
  arXiv  Detail & Related papers  (2026-05-30T12:07:32Z)
• JFAA: Technical Report for the EPIC-KITCHENS-100 Action Anticipation Challenge at EgoVis 2026 [70.67609435035631]
  JFAA is a JEPA-based Future Action Anticipation method for the EPIC-KITCHENS-100 (EK-100) Action Anticipation task.<n>Inspired by the representation learning and future prediction ability of V-JEPA 2.1, JFAA uses a frozen encoder and predictor to extract observed context features.<n>A lightweight attentive probe is then trained to predict verb, noun, and action logits with separate task queries.
  arXiv  Detail & Related papers  (2026-05-20T08:47:50Z)
• VISTA: Technical Report for the Ego4D Short-Term Object Interaction Anticipation at EgoVis 2026 [70.67609435035631]
  Given an egocentric video timestamp, the task requires anticipating the next human-object interaction.<n>ViSTA follows a StillFast-style design that combines object-centric spatial detection with short-horizon temporal context.<n>Experiments show that VISTA achieves first place in the EgoVis 2026 Ego4D STA Challenge.
  arXiv  Detail & Related papers  (2026-05-20T08:42:56Z)
• NVIDIA-UNIBZ Submission for EPIC-KITCHENS-100 Action Anticipation Challenge 2022 [13.603712913129506]
  We describe the technical details of our submission for the EPIC-Kitchen-100 action anticipation challenge. Our modelings, the higher-order recurrent space-time transformer and the message-passing neural network with edge learning, are both recurrent-based architectures which observe only 2.5 seconds inference context to form the action anticipation prediction. By averaging the prediction scores from a set of models compiled with our proposed training pipeline, we achieved strong performance on the test set, which is 19.61% overall mean top-5 recall, recorded as second place on the public leaderboard.
  arXiv  Detail & Related papers  (2022-06-22T06:34:58Z)
• End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillation [86.41437210485932]
  We aim at advancing zero-shot HOI detection to detect both seen and unseen HOIs simultaneously. We propose a novel end-to-end zero-shot HOI Detection framework via vision-language knowledge distillation. Our method outperforms the previous SOTA by 8.92% on unseen mAP and 10.18% on overall mAP.
  arXiv  Detail & Related papers  (2022-04-01T07:27:19Z)
• Team Enigma at ArgMining-EMNLP 2021: Leveraging Pre-trained Language Models for Key Point Matching [0.0]
  We present the system description for our submission towards the Key Point Analysis Shared Task at ArgMining 2021. We leveraged existing state of the art pre-trained language models along with incorporating additional data and features extracted from the inputs (topics, key points, and arguments) to improve performance. We were able to achieve mAP strict and mAP relaxed score of 0.872 and 0.966 respectively in the evaluation phase, securing 5th place on the leaderboard.
  arXiv  Detail & Related papers  (2021-10-24T07:10:39Z)
• SAIC_Cambridge-HuPBA-FBK Submission to the EPIC-Kitchens-100 Action Recognition Challenge 2021 [80.05652375838073]
  This report presents the technical details of our submission to the EPIC-Kitchens-100 Action Recognition Challenge 2021. Our submission, visible on the public leaderboard, achieved a top-1 action recognition accuracy of 44.82%, using only RGB.
  arXiv  Detail & Related papers  (2021-10-06T16:29:47Z)
• TransAction: ICL-SJTU Submission to EPIC-Kitchens Action Anticipation Challenge 2021 [42.35018041385645]
  We developed a hierarchical attention model for action anticipation. In terms of Mean Top-5 Recall of action, our submission with team name ICL-SJTU achieved 13.39%. It is noteworthy that our submission ranked 1st in terms of verb class in all three (sub)sets.
  arXiv  Detail & Related papers  (2021-07-28T10:42:47Z)
• Anticipative Video Transformer [105.20878510342551]
  Anticipative Video Transformer (AVT) is an end-to-end attention-based video modeling architecture. We train the model jointly to predict the next action in a video sequence, while also learning frame feature encoders that are predictive of successive future frames' features.
  arXiv  Detail & Related papers  (2021-06-03T17:57:55Z)
• Rescaling Egocentric Vision [48.57283024015145]
  This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos. Compared to its previous version, EPIC-KITCHENS-100 has been annotated using a novel pipeline that allows denser (54% more actions per minute) and more complete annotations of fine-grained actions (+128% more action segments)
  arXiv  Detail & Related papers  (2020-06-23T18:28:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.