EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Grounding

• URL: http://arxiv.org/abs/2605.14742v1
• Date: Thu, 14 May 2026 12:10:27 GMT
• Title: EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Grounding
• Abstract summary: This paper introduces EARL, an Egocentric Analysis-guided Reinforcement Learning framework that transfers coarse interaction semantics to query-oriented answering and grounding.<n>Specifically, EARL adopts a two-stage parsing framework including coarse-grained interpretation and fine-grained response.<n> Experiments on Ego-IRGBench show that EARL achieves 65.48% cIoU for pixel grounding, outperforming previous RL-based methods by 8.37%.
• Score: 21.745254094130075
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding human--environment interactions from egocentric vision is essential for assistive robotics and embodied intelligent agents, yet existing multimodal large language models (MLLMs) still struggle with accurate interaction reasoning and fine-grained pixel grounding. To this end, this paper introduces EARL, an Egocentric Analysis-guided Reinforcement Learning framework that explicitly transfers coarse interaction semantics to query-oriented answering and grounding. Specifically, EARL adopts a two-stage parsing framework including coarse-grained interpretation and fine-grained response. The first stage holistically interprets egocentric interactions and generates a structured textual description. The second stage produces the textual answer and pixel-level mask in response to the user query. To bridge the two stages, we extract a global interaction descriptor as a semantic prior, which is integrated via a novel Analysis-guided Feature Synthesizer (AFS) for query-oriented reasoning. To optimize heterogeneous outputs, including textual answers, bounding boxes, and grounding masks, we design a multi-faceted reward function and train the response stage with GRPO. Experiments on Ego-IRGBench show that EARL achieves 65.48% cIoU for pixel grounding, outperforming previous RL-based methods by 8.37%, while OOD grounding results on EgoHOS indicate strong transferability to unseen egocentric grounding scenarios.

Related papers

• Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild [49.77540427384148]
  We present AgentHOI, a training-free, agentic framework that transfers the generalist multimodal reasoning capabilities of foundation models to HOI detection in the wild.<n>To address the challenges of incomplete interaction discovery and ambiguous localization in complex scenes, we introduce two key mechanisms.<n>AgentHOI achieves superior performance over state-of-the-art supervised and weakly supervised methods in real-world settings.
  arXiv  Detail & Related papers  (2026-07-15T14:30:20Z)
• Decoupling Semantics and Logic: A Training-Free Coarse-to-Fine Pipeline for Video Retrieval-Augmented Generation [5.263138429758972]
  This paper presents our system description for the 2nd Workshop on Multimodal Augmented Generation via MultimodAl Retrieval (MAGMaR)<n> Addressing the critical challenges of cross-lingual long-video comprehension, strict persona adherence, and zero-hallucination temporal grounding, we propose a two-stage cascaded Video RAG pipeline.<n>Our architecture strategically decouples semantic retrieval from cognitive logical reasoning through a modality-aware division of labor.
  arXiv  Detail & Related papers  (2026-06-06T01:17:40Z)
• Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Vision [13.21187394955871]
  EgoPoint-Thought is the first large-scale multimodal dataset dedicated to egocentric deictic visual grounding.<n>It provides rich, multi-grained annotations including hand-target bounding box pairs and dense semantic captions.<n>We propose SV-CoT, a novel baseline framework that reformulates grounding as a structured inference process.
  arXiv  Detail & Related papers  (2026-03-27T17:49:56Z)
• Ego to World: Collaborative Spatial Reasoning in Embodied Systems via Reinforcement Learning [61.753025885751036]
  We introduce the Ego-to-World benchmark, which evaluates a vision-language model's ability to fuse heterogeneous viewpoints across three tasks.<n>We propose CoRL, a two-stage framework that combines Chain-of-Thought supervised fine-tuning with reinforcement learning.<n>We show that CoRL consistently surpasses strong proprietary and open-source baselines on both reasoning and perception-grounding metrics.
  arXiv  Detail & Related papers  (2026-03-16T04:27:53Z)
• Synergizing Understanding and Generation with Interleaved Analyzing-Drafting Thinking [154.2388970262703]
  Unified Vision-Language Models (UVLMs) aim to advance multimodal learning by supporting both understanding and generation within a single framework.<n>We introduce the interleaved Analyzing-Drafting problem-solving loop (AD-Loop), a new think paradigm that alternates between analytic and drafting operations.<n>By interleaving textual thoughts with visual thoughts, AD-Loop enables models to iteratively refine both comprehension and outputs, fostering genuine synergy.
  arXiv  Detail & Related papers  (2026-02-24T23:26:09Z)
• Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention [58.05340906967343]
  Egocentric Referring Video Object (Ego-RVOS) aims to segment the specific object actively involved in a human action, as described by a language query, within first-person videos.<n>Existing methods often struggle, learning spurious correlations from skewed object-action pairings in datasets.<n>We introduce Causal-REferring (CERES), a plug-in causal framework that adapts strong, pre-trained RVOSs to the egocentric domain.
  arXiv  Detail & Related papers  (2025-12-30T16:22:14Z)
• Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning [63.109585527799005]
  GroundingAgent is a visual grounding framework that operates without task-specific fine-tuning.<n>It achieves an average zero-shot grounding accuracy of 65.1 % on widely-used benchmarks.<n>It also offers strong interpretability, transparently illustrating each reasoning step.
  arXiv  Detail & Related papers  (2025-11-24T03:11:08Z)
• ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interaction [16.338872733140832]
  This paper presents a novel task named Egocentric Interaction Reasoning and pixel Grounding (Ego-IRG)<n>Taking an egocentric image with the query as input, Ego-IRG is the first task that aims to resolve the interactions through three crucial steps: analyzing, answering, and pixel grounding.<n>The Ego-IRGBench dataset includes over 20k egocentric images with 1.6 million queries and corresponding multimodal responses about interactions.
  arXiv  Detail & Related papers  (2025-04-02T08:24:35Z)
• Iteration of Thought: Leveraging Inner Dialogue for Autonomous Large Language Model Reasoning [0.0]
  Iterative human engagement is a common and effective means of leveraging the advanced language processing power of large language models (LLMs) We propose the Iteration of Thought (IoT) framework for enhancing LLM responses by generating "thought"-provoking prompts. Unlike static or semi-static approaches, IoT adapts its reasoning path dynamically, based on evolving context.
  arXiv  Detail & Related papers  (2024-09-19T09:44:17Z)
• Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models [55.20626448358655]
  This study explores the universal interaction recognition in an open-world setting through the use of Vision-Language (VL) foundation models and large language models (LLMs) Our design includes an HO Prompt-guided Decoder (HOPD), facilitates the association of high-level relation representations in the foundation model with various HO pairs within the image. For open-category interaction recognition, our method supports either of two input types: interaction phrase or interpretive sentence.
  arXiv  Detail & Related papers  (2023-11-07T08:27:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.