Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos
- URL: http://arxiv.org/abs/2603.22529v1
- Date: Mon, 23 Mar 2026 19:47:58 GMT
- Title: Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos
- Abstract summary: Ego2Web is the first benchmark designed to bridge egocentric video perception and web agent execution.<n>It pairs real-world first-person video recordings with web tasks that require visual understanding, web task planning, and interaction in an online environment.<n>We utilize an automatic data-generation pipeline combined with human verification and refinement to curate well-constructed, high-quality video-task pairs.
- Score: 74.78784171851984
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Multimodal AI agents are increasingly automating complex real-world workflows that involve online web execution. However, current web-agent benchmarks suffer from a critical limitation: they focus entirely on web-based interaction and perception, lacking grounding in the user's real-world physical surroundings. This limitation prevents evaluation in crucial scenarios, such as when an agent must use egocentric visual perception (e.g., via AR glasses) to recognize an object in the user's surroundings and then complete a related task online. To address this gap, we introduce Ego2Web, the first benchmark designed to bridge egocentric video perception and web agent execution. Ego2Web pairs real-world first-person video recordings with web tasks that require visual understanding, web task planning, and interaction in an online environment for successful completion. We utilize an automatic data-generation pipeline combined with human verification and refinement to curate well-constructed, high-quality video-task pairs across diverse web task types, including e-commerce, media retrieval, knowledge lookup, etc. To facilitate accurate and scalable evaluation for our benchmark, we also develop a novel LLM-as-a-Judge automatic evaluation method, Ego2WebJudge, which achieves approximately 84% agreement with human judgment, substantially higher than existing evaluation methods. Experiments with diverse SoTA agents on our Ego2Web show that their performance is weak, with substantial headroom across all task categories. We also conduct a comprehensive ablation study on task design, highlighting the necessity of accurate video understanding in the proposed task and the limitations of current agents. We hope Ego2Web can be a critical new resource for developing truly capable AI assistants that can seamlessly see, understand, and act across the physical and digital worlds.
Related papers
- Egocentric Co-Pilot: Web-Native Smart-Glasses Agents for Assistive Egocentric AI [56.98603185789977]
We present Egocentric Co-Pilot, a web-native neuro-symbolic framework that runs on smart glasses.<n>We use a Large Language Model (LLM) to orchestrate a toolbox of perception, reasoning, and web tools.<n> Experiments on Egolife and HD-EPIC demonstrate competitive or state-of-the-art egocentric QA performance.
arXiv Detail & Related papers (2026-03-01T13:43:04Z) - Cognitive Duality for Adaptive Web Agents [3.0069922338220825]
We propose a principled decomposition into fast System 1 and slow System 2 cognitive processes.<n>We implement this framework in CogniWeb, a modular agent architecture that adaptively toggles between fast intuitive processing and deliberate reasoning based on task complexity.
arXiv Detail & Related papers (2025-08-07T07:05:22Z) - Embodied Web Agents: Bridging Physical-Digital Realms for Integrated Agent Intelligence [109.32705135051486]
Embodied Web Agents is a novel paradigm for AI agents that fluidly bridge the embodiment and web-scale reasoning.<n>We release the Embodied Web Agents Benchmark, which encompasses a diverse suite of tasks.<n>Results reveal significant performance gaps between state-of-the-art AI systems and human capabilities.
arXiv Detail & Related papers (2025-06-18T17:58:17Z) - An Illusion of Progress? Assessing the Current State of Web Agents [61.742657650092845]
We conduct a comprehensive and rigorous assessment of the current state of web agents.<n>Results depict a very different picture of the competency of current agents, suggesting over-optimism in previously reported results.<n>We introduce Online-Mind2Web, an online evaluation benchmark consisting of 300 diverse and realistic tasks spanning 136 websites.
arXiv Detail & Related papers (2025-04-02T05:51:29Z) - TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks [55.03911355902567]
We introduce TheAgentCompany, a benchmark for evaluating AI agents that interact with the world in similar ways to those of a digital worker.<n>We find that the most competitive agent can complete 30% of tasks autonomously.<n>This paints a nuanced picture on task automation with simulating LM agents in a setting a real workplace.
arXiv Detail & Related papers (2024-12-18T18:55:40Z) - Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents [64.75036903373712]
Proposer-Agent-Evaluator is a learning system that enables foundation model agents to autonomously discover and practice skills in the wild.<n>At the heart of PAE is a context-aware task proposer that autonomously proposes tasks for the agent to practice with context information.<n>The success evaluation serves as the reward signal for the agent to refine its policies through RL.
arXiv Detail & Related papers (2024-12-17T18:59:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.