EgoProceVQA: A Novel Egocentric Procedural Understanding Task with Self-Skill-Exploration Agent
- URL: http://arxiv.org/abs/2607.13792v1
- Date: Wed, 15 Jul 2026 12:55:35 GMT
- Title: EgoProceVQA: A Novel Egocentric Procedural Understanding Task with Self-Skill-Exploration Agent
- Abstract summary: Egocentric Procedural Understanding VQA task (EgoProceVQA)<n>EgoProceGen, a data generation platform that efficiently constructs QA data tailored to different question types.<n>EgoProceAgent, a self-skill-exploration agentic framework.
- Score: 40.15272756472325
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Most daily activities are inherently procedural. However, existing evaluations for egocentric video understanding seldom address procedural understanding and largely overlook complex key-step-level reasoning under the widely used video question answering (VQA) paradigm for MLLMs. Such capabilities are crucial for building procedural AI assistants deployable on wearable devices. To bridge this gap, we introduce the Egocentric Procedural Understanding VQA task (EgoProceVQA), which systematically evaluates egocentric procedural reasoning abilities of current MLLMs and agents through six types of key-step-centric questions. Furthermore, we develop EgoProceGen, a data generation platform that efficiently constructs QA data tailored to different question types. Based on this platform, we build a benchmark with 3,600 questions, four common procedural scenarios, and 31 everyday procedural tasks. Evaluations on EgoProceVQA show that existing MLLMs and agents still have substantial room for improvement in procedural understanding. Therefore, we further propose EgoProceAgent, a self-skill-exploration agentic framework. We design a generic tool library for procedural understanding and a standardized sub-skill library shared across tools and models, enabling self-exploration without ground-truth supervision. By exploring how to compose and select sub-skills, the agent discovers effective skill strategies for diverse problems, and attains state-of-the-art performance among open-source models on multiple tasks. Together, our benchmark, generation platform, and agentic framework establish a unified foundation for EgoProceVQA. Project page: https://z1oong.github.io/EgoProceVQA/.
Related papers
- EgoErrorVQA: Assess Egocentric Comprehension Capabilities through Procedural Errors for Ego-Agentic AI [24.612087629174656]
EgoErrorVQA task is proposed for egocentric procedural comprehension with explicit procedural errors modeling.<n>We develop a user-friendly evaluator agent based on the Agent2Agent protocol.<n>We also introduce Ego-ADR, an Adaptive Decoupled Reasoning framework.
arXiv Detail & Related papers (2026-08-25T06:58:44Z) - From Knowing to Acting: Benchmarking Self-Awareness Capability of LLM Agents [69.07911499189129]
We introduce KAPRO, a framework that evaluates cognitive-behavioral alignment by decoupling an agent's metacognitive judgment (Knowing) from its spontaneous execution (Acting)<n>Experiments show that self-awareness capability is strongly correlated with task success but degrades sharply in internal-capability settings.<n> proprietary and reasoning-oriented models demonstrate more reliable cognitive gating.
arXiv Detail & Related papers (2026-06-09T17:17:08Z) - MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents [54.48066948369172]
As embodied models become powerful, humans will collaborate with multiple embodied AI agents at their workplace or home in the future.<n>Existing challenges include effectively compressing and communicating high volumes of individual sensory inputs in the form of video.<n>We first formally define a novel problem of understanding multiple long-horizon egocentric videos simultaneously collected from embodied agents.
arXiv Detail & Related papers (2026-03-10T15:48:35Z) - ProMQA-Assembly: Multimodal Procedural QA Dataset on Assembly [13.040491675077687]
We propose a new multimodal QA dataset on assembly activities.<n>Our dataset, ProMQA-Assembly, consists of 391 QA pairs that require the multimodal understanding of human-activity recordings and their instruction manuals.
arXiv Detail & Related papers (2025-09-03T02:26:48Z) - Memento No More: Coaching AI Agents to Master Multiple Tasks via Hints Internalization [56.674356045200696]
We propose a novel method to train AI agents to incorporate knowledge and skills for multiple tasks without the need for cumbersome note systems or prior high-quality demonstration data.<n>Our approach employs an iterative process where the agent collects new experiences, receives corrective feedback from humans in the form of hints, and integrates this feedback into its weights.<n>We demonstrate the efficacy of our approach by implementing it in a Llama-3-based agent that, after only a few rounds of feedback, outperforms advanced models GPT-4o and DeepSeek-V3 in tasksets.
arXiv Detail & Related papers (2025-02-03T17:45:46Z) - TaskBench: Benchmarking Large Language Models for Task Automation [82.2932794189585]
We introduce TaskBench, a framework to evaluate the capability of large language models (LLMs) in task automation.
Specifically, task decomposition, tool selection, and parameter prediction are assessed.
Our approach combines automated construction with rigorous human verification, ensuring high consistency with human evaluation.
arXiv Detail & Related papers (2023-11-30T18:02:44Z) - Attributed Question Answering: Evaluation and Modeling for Attributed
Large Language Models [68.37431984231338]
Large language models (LLMs) have shown impressive results across a variety of tasks while requiring little or no direct supervision.
We believe the ability of an LLM to an attribute to the text that it generates is likely to be crucial for both system developers and users in this setting.
arXiv Detail & Related papers (2022-12-15T18:45:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.