Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning

• URL: http://arxiv.org/abs/2605.13335v1
• Date: Wed, 13 May 2026 10:53:15 GMT
• Title: Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning
• Abstract summary: We introduce Ego2World, an executable benchmark that turns egocentric cooking videos into executable symbolic worlds governed by graph-transition rules.<n>During evaluation, the simulator maintains a hidden world graph, while the agent plans over its own partial belief graph using only local observations and execution feedback.<n>This separation forces agents to update memory and replan without observing the true world state.
• Score: 55.43343782036886
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Embodied agents in household environments must plan under partial observation: they need to remember objects, track state changes, and recover when actions fail. Existing benchmarks only partially test this ability. Egocentric video datasets capture realistic human activities but remain passive, while interactive simulators support execution but rely on synthetic scenes and hand-crafted dynamics, introducing a sim-to-real gap and often assuming fully observable state. We introduce Ego2World, an executable benchmark that turns egocentric cooking videos into executable symbolic worlds governed by graph-transition rules. Built on HD-EPIC, Ego2World derives reusable transition rules from video annotations and executes them in a hidden symbolic world graph. During evaluation, the simulator maintains the hidden world graph, while the agent plans over its own partial belief graph using only local observations and execution feedback. This separation forces agents to update memory and replan without observing the true world state. Experiments show that action-overlap scores overestimate physical-state success, and that persistent belief memory improves task completion while reducing repeated visual exploration -- suggesting that belief maintenance should be a first-class target of embodied-agent evaluation.

Related papers

• Towards Generalizable Visually Grounded Exploration of Household Devices [74.66117923766632]
  Key bottleneck of realizing general autonomous embodied agents lies in Generalizable Visually Grounded Exploration.<n>We introduce VGEBench, a benchmark designed to evaluate the generalizable visually grounded exploration capabilities of Vision-Language Models.<n>Unlike static datasets, we construct a Logic-Driven State Machine framework. This framework simulates multi-turn interaction loops, compelling agents to achieve goals by active visual perception and feedback-driven correction.
  arXiv  Detail & Related papers  (2026-09-01T07:42:45Z)
• EgoExo-WM: Unlocking Exo Video for Ego World Models [59.337519140691775]
  Egocentric world models present a promising direction for enabling agents to predict and plan.<n>Exocentric video is abundant and reveals body poses well, but lacks direct alignment with an agent's action space.<n>We propose a method to bridge this gap by extracting structured body pose from exocentric video as a representation of action.
  arXiv  Detail & Related papers  (2026-05-14T23:35:54Z)
• EgoSim: Egocentric World Simulator for Embodied Interaction Generation [93.11209644808783]
  EgoSim is a closed-loop egocentric world simulator that generates spatially consistent interaction videos.<n>It persistently updates the underlying 3D scene state for continuous simulation.<n>EgoSim significantly outperforms existing methods in terms of visual quality, spatial consistency, and generalization.
  arXiv  Detail & Related papers  (2026-04-01T15:00:46Z)
• EgoForge: Goal-Directed Egocentric World Simulator [26.712565464146937]
  EgoForge is a goal-directed world simulator that generates coherent, first-person video rollouts from minimal static inputs.<n>VideoDiffusionNFT is a trajectory-level reward-guided refinement that optimize goal completion, temporal causality, scene consistency, and perceptual fidelity during diffusion sampling.
  arXiv  Detail & Related papers  (2026-03-20T17:46:55Z)
• Self-Correcting VLA: Online Action Refinement via Sparse World Imagination [55.982504915794514]
  We propose Self-Correcting VLA (SC-VLA), which achieve self-improvement by intrinsically guiding action refinement through sparse imagination.<n>SC-VLA achieve state-of-the-art performance, yielding the highest task throughput with 16% fewer steps and a 9% higher success rate than the best-performing baselines.
  arXiv  Detail & Related papers  (2026-02-25T06:58:06Z)
• EgoPush: Learning End-to-End Egocentric Multi-Object Rearrangement for Mobile Robots [13.858983945750081]
  EgoPush is a policy learning framework that enables egocentric, perception-driven rearrangement.<n>We show that EgoPush significantly outperforms end-to-end RL baselines in success rate.<n>We also demonstrate zero-shot sim-to-real transfer on a mobile platform in the real world.
  arXiv  Detail & Related papers  (2026-02-20T08:54:20Z)
• ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction [35.24704057622881]
  Embodied cognition argues that intelligence arises from sensorimotor interaction rather than passive observation.<n>We introduce ENACT, a benchmark that casts evaluation of embodied cognition as world modeling from egocentric interaction.
  arXiv  Detail & Related papers  (2025-11-26T00:06:02Z)
• PlayerOne: Egocentric World Simulator [73.88786358213694]
  PlayerOne is the first egocentric realistic world simulator.<n>It generates egocentric videos that are strictly aligned with the real scene human motion of the user captured by an exocentric camera.
  arXiv  Detail & Related papers  (2025-06-11T17:59:53Z)
• EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real World [12.699670048897085]
  In human imitation learning, the imitator typically take the egocentric view as a benchmark, naturally transferring behaviors observed from an exocentric view to their owns.<n>We introduce EgoMe, which towards following the process of human imitation learning via the imitator's egocentric view in the real world.<n>Our dataset includes 7902 paired exo-ego videos spanning diverse daily behaviors in various real-world scenarios.
  arXiv  Detail & Related papers  (2025-01-31T11:48:22Z)
• WorldSimBench: Towards Video Generation Models as World Simulators [79.69709361730865]
  We classify the functionalities of predictive models into a hierarchy and take the first step in evaluating World Simulators by proposing a dual evaluation framework called WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, encompassing human preference assessments from the visual perspective and action-level evaluations in embodied tasks. Our comprehensive evaluation offers key insights that can drive further innovation in video generation models, positioning World Simulators as a pivotal advancement toward embodied artificial intelligence.
  arXiv  Detail & Related papers  (2024-10-23T17:56:11Z)
• EgoGen: An Egocentric Synthetic Data Generator [53.32942235801499]
  EgoGen is a new synthetic data generator that can produce accurate and rich ground-truth training data for egocentric perception tasks. At the heart of EgoGen is a novel human motion synthesis model that directly leverages egocentric visual inputs of a virtual human to sense the 3D environment. We demonstrate EgoGen's efficacy in three tasks: mapping and localization for head-mounted cameras, egocentric camera tracking, and human mesh recovery from egocentric views.
  arXiv  Detail & Related papers  (2024-01-16T18:55:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.