VISTA: A Generative Egocentric Video Framework for Daily Assistance

• URL: http://arxiv.org/abs/2605.10579v1
• Date: Mon, 11 May 2026 13:50:47 GMT
• Title: VISTA: A Generative Egocentric Video Framework for Daily Assistance
• Abstract summary: We introduce VISTA, a video synthesis system that produces high-fidelity egocentric videos as training and evaluation data for AI agents.<n> VISTA employs a 5-step script generation pipeline with causal reverse reasoning to create diverse, logically grounded intervention modes.<n>These scenarios span two levels of agent autonomy: reactive and proactive.
• Score: 5.919102792488079
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training AI agents to proactively assist humans in daily activities, from routine household tasks to urgent safety situations, requires large-scale visual data. However, capturing such scenarios in the real world is often difficult, costly, or unsafe, and physics-based simulators lack the visual fidelity needed to transfer learned behaviors to real settings. Therefore, we introduce VISTA, a video synthesis system that produces high-fidelity egocentric videos as training and evaluation data for AI agents. VISTA employs a 5-step script generation pipeline with causal reverse reasoning to create diverse, logically grounded intervention modes. These scenarios span two levels of agent autonomy: reactive and proactive. In reactive modes, the user explicitly asks the agent for help. In proactive modes, the agent offers help without receiving a direct request. We further divide proactive modes into explicit and implicit types. In explicit proactive scenarios, the user is aware of needing help but does not directly address the agent. In implicit proactive scenarios, the agent intervenes before the user even realizes that help is needed. VISTA allows users to customize and refine scenarios to generate video benchmarks for daily tasks, offering a scalable and controllable alternative to real-world data collection for training and evaluating AI agents in realistic environments.

Related papers

• Time-Aware Assistive Navigation [17.665122992601788]
  Current language models rarely plan over whether and when to realize a real-time response to a user.<n>We introduce a large-scale benchmark for an egocentric, assistive navigation task in complex outdoor environments.<n>We show that a simple yet effective modification of the model, including direct supervision to predict the underlying reason for each instruction, yields significant performance gains.
  arXiv  Detail & Related papers  (2026-09-04T17:59:47Z)
• SAGE: Scalable Agentic 3D Scene Generation for Embodied AI [67.43935343696982]
  Existing scene-generation systems often rely on rule-based or task-specific pipelines, yielding artifacts and physically invalid scenes.<n>We present SAGE, an agentic framework that, given a user-specified embodied task, understands the intent and automatically generates simulation-ready environments at scale.<n>The resulting environments are realistic, diverse, and directly deployable in modern simulators for policy training.
  arXiv  Detail & Related papers  (2026-02-10T18:59:55Z)
• What's the next frontier for Data-centric AI? Data Savvy Agents [71.76058707995398]
  We argue that data-savvy capabilities should be a top priority in the design of agentic systems.<n>We propose four key capabilities to realize this vision: Proactive data acquisition, Sophisticated data processing, Interactive test data synthesis, and Continual adaptation.
  arXiv  Detail & Related papers  (2025-11-02T17:09:29Z)
• Dyna-Mind: Learning to Simulate from Experience for Better AI Agents [62.21219817256246]
  We argue that current AI agents need ''vicarious trial and error'' - the capacity to mentally simulate alternative futures before acting.<n>We introduce Dyna-Mind, a two-stage training framework that explicitly teaches (V)LM agents to integrate such simulation into their reasoning.
  arXiv  Detail & Related papers  (2025-10-10T17:30:18Z)
• Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents [58.00130492861884]
  TraitBasis is a lightweight, model-agnostic method for systematically stress testing AI agents.<n>TraitBasis learns directions in activation space corresponding to steerable user traits.<n>We observe on average a 2%-30% performance degradation on $tau$-Trait across frontier models.
  arXiv  Detail & Related papers  (2025-10-06T05:03:57Z)
• AirVista-II: An Agentic System for Embodied UAVs Toward Dynamic Scene Semantic Understanding [16.405658563770757]
  AirVista-II is an end-to-end agentic system for embodied UAVs.<n>System integrates agent-based task identification and scheduling, multimodal perception mechanisms, and differentiated extraction strategies.
  arXiv  Detail & Related papers  (2025-04-13T14:06:50Z)
• Reasoning in visual navigation of end-to-end trained agents: a dynamical systems approach [23.52028824411467]
  We present a large-scale experimental study involving numepisodes navigation episodes in a real environment with a physical robot.<n>We analyze the type of reasoning emerging from end-to-end training.<n>We show in a post-hoc analysis that the value function learned by the agent relates to long-term planning.
  arXiv  Detail & Related papers  (2025-03-11T11:16:47Z)
• Memento No More: Coaching AI Agents to Master Multiple Tasks via Hints Internalization [56.674356045200696]
  We propose a novel method to train AI agents to incorporate knowledge and skills for multiple tasks without the need for cumbersome note systems or prior high-quality demonstration data.<n>Our approach employs an iterative process where the agent collects new experiences, receives corrective feedback from humans in the form of hints, and integrates this feedback into its weights.<n>We demonstrate the efficacy of our approach by implementing it in a Llama-3-based agent that, after only a few rounds of feedback, outperforms advanced models GPT-4o and DeepSeek-V3 in tasksets.
  arXiv  Detail & Related papers  (2025-02-03T17:45:46Z)
• YETI (YET to Intervene) Proactive Interventions by Multimodal AI Agents in Augmented Reality Tasks [16.443149180969776]
  Augmented Reality (AR) head worn devices can uniquely improve the user experience of solving procedural day-to-day tasks.<n>Such AR capabilities can help AI Agents see and listen to actions that users take which can relate to multimodal capabilities of human users.<n>Proactivity of AI Agents on the other hand can help the human user detect and correct any mistakes in agent observed tasks.
  arXiv  Detail & Related papers  (2025-01-16T08:06:02Z)
• Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialogues [54.81155589931697]
  Collaborative Instance object Navigation (CoIN) is a new task setting where the agent actively resolve uncertainties about the target instance.<n>We propose a novel training-free method, Agent-user Interaction with UncerTainty Awareness (AIUTA)<n>First, upon object detection, a Self-Questioner model initiates a self-dialogue within the agent to obtain a complete and accurate observation description.<n>An Interaction Trigger module determines whether to ask a question to the human, continue or halt navigation.
  arXiv  Detail & Related papers  (2024-12-02T08:16:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.