OSGNet with MLLM Reranking @ Ego4D Episodic Memory Challenge 2026

• URL: http://arxiv.org/abs/2605.20818v1
• Date: Wed, 20 May 2026 07:14:11 GMT
• Title: OSGNet with MLLM Reranking @ Ego4D Episodic Memory Challenge 2026
• Abstract summary: We present our champion solutions for the Natural Language Queries and GoalStep tracks of the Ego4D Episodic Memory Challenge at CVPR 2026.<n>We propose a reranking-based framework that effectively leverages the strong video-language reasoning capability of multimodal large language model (MLLM)<n>Our method achieved first place in both the Natural Language Queries and GoalStep tracks.
• Score: 75.6762616797288
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this report, we present our champion solutions for the Natural Language Queries and GoalStep tracks of the Ego4D Episodic Memory Challenge at CVPR 2026. Both tracks require accurately localizing temporal segments from long untrimmed egocentric videos. To address these tasks, we propose a reranking-based framework that effectively leverages the strong video-language reasoning capability of multimodal large language model (MLLM) while preserving the efficiency and candidate recall of conventional localization pipelines. Specifically, we first obtain a set of candidate segments from existing localization model OSGNet, and then employ MLLM to select the segment that best matches the given query, thereby refining the final prediction. Ultimately, our method achieved first place in both the Natural Language Queries and GoalStep tracks. Our code can be found at https://github.com/iLearn-Lab/CVPR25-OSGNet.

Related papers

• Multilingual Long-Form Speech Instruction Following: KIT's Submission to IWSLT 2026 [57.48615919717867]
  We present KIT's submission to the Long and Short Instruction Following tracks in the unconstrained setting.<n>Our approach combines a general data augmentation pipeline that converts short-form corpora into long-form training data.<n>We show that likelihood-based re-ranking, while highly effective for ASR, systematically degrades semantic tasks.
  arXiv  Detail & Related papers  (2026-06-03T11:13:37Z)
• OSGNet @ Ego4D Episodic Memory Challenge 2025 [77.414837862995]
  We present our champion solutions for the three egocentric video localization tracks of the Ego4D Episodic Memory Challenge at CVPR 2025.<n>We adopt an early fusion-based video localization model to tackle all three tasks, aiming to enhance localization accuracy.
  arXiv  Detail & Related papers  (2025-06-04T08:41:42Z)
• Segment First or Comprehend First? Explore the Limit of Unsupervised Word Segmentation with Large Language Models [92.92512796044471]
  We propose a new framework to explore the limit of unsupervised word segmentation with Large Language Models (LLMs)<n>We employ current mainstream LLMs to perform word segmentation across multiple languages to assess LLMs' "comprehension"<n>We introduce a novel unsupervised method, termed LLACA, which enables the construction of a dynamic $n$-gram model that adjusts based on contextual information.
  arXiv  Detail & Related papers  (2025-05-26T07:48:15Z)
• Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models [7.213221003652941]
  This paper proposes Moment-GPT, a tuning-free pipeline for zero-shot VMR utilizing frozen MLLMs.<n>We first employ LLaMA-3 to correct and rephrase the query to mitigate language bias. Subsequently, we design a span generator combined with MiniGPT-v2 to produce candidate spans adaptively.<n>Our proposed method substantially outperforms the state-ofthe-art MLLM-based and zero-shot models on several public datasets.
  arXiv  Detail & Related papers  (2025-01-14T09:45:10Z)
• ObjectNLQ @ Ego4D Episodic Memory Challenge 2024 [51.57555556405898]
  We present our approach for the Natural Language Query track and Goal Step track of the Ego4D Episodic Memory Benchmark at CVPR 2024. Both challenges require the localization of actions within long video sequences using textual queries. We introduce a novel approach, termed ObjectNLQ, which incorporates an object branch to augment the video representation with detailed object information.
  arXiv  Detail & Related papers  (2024-06-22T07:57:58Z)
• LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoning [8.379286663107845]
  Reasoning segmentation is a novel task that enables segmentation system to reason and interpret implicit user intention. Our work on reasoning segmentation contributes on both the methodological design and dataset labeling.
  arXiv  Detail & Related papers  (2024-04-12T18:45:51Z)
• Empowering Segmentation Ability to Multi-modal Large Language Models [19.685984211930354]
  We extend multi-modal large language models (MLLMs) with the segmentation ability. The extended MLLMs can both output language responses to the image-language prompts and segment the regions that the complex question or query in the language prompts focuses on. We propose a novel MLLMs framework, coined as LLaVASeg, which leverages a chain-of-thought prompting strategy to instruct the MLLMs to segment the target region queried by the user.
  arXiv  Detail & Related papers  (2024-03-21T05:36:25Z)
• The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics [74.99898531299148]
  This research examines vocabulary trimming (VT) inspired by restricting embedding entries to the language of interest to bolster time and memory efficiency. We apply two languages to trim the full vocabulary - Unicode-based script filtering and corpus-based selection - to different language families and sizes. It is found that VT reduces the memory usage of small models by nearly 50% and has an upper bound of 25% improvement in generation speed.
  arXiv  Detail & Related papers  (2023-11-16T09:35:50Z)
• Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models [30.48902594738911]
  Given a long conversation, large language models (LLMs) fail to recall past information and tend to generate inconsistent responses.<n>We propose to generate summaries/ memory using large language models (LLMs) to enhance long-term memory ability.
  arXiv  Detail & Related papers  (2023-08-29T04:59:53Z)
• Weakly-Supervised Temporal Action Localization Through Local-Global Background Modeling [30.104982661371164]
  We present our 2021 HACS Challenge - Weakly-supervised Learning Track solution that based on BaSNet to address above problem. Specifically, we first adopt pre-trained CSN, Slowfast, TDN, and ViViT as feature extractors to get feature sequences. Then our proposed Local-Global Background Modeling Network (LGBM-Net) is trained to localize instances by using only video-level labels.
  arXiv  Detail & Related papers  (2021-06-20T02:58:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.