Diagnosing Semantic Handoff Failures in Agent-Orchestrated Vision-Language-Action Skill Composition

• URL: http://arxiv.org/abs/2607.06256v2
• Date: Wed, 15 Jul 2026 12:10:06 GMT
• Title: Diagnosing Semantic Handoff Failures in Agent-Orchestrated Vision-Language-Action Skill Composition
• Abstract summary: Long-horizon household tasks require robots to compose many language-conditioned skills.<n>A skill may satisfy its own postcondition while leaving the robot in a state from which the next skill cannot reliably start.<n>We study this semantic handoff problem in BEHAVIOR-1K through an agent-orchestrated vision-language-action execution harness.
• Score: 10.276503485743111
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-horizon household tasks require robots to compose many language-conditioned skills, yet the boundary between consecutive skills is rarely explicit. A skill may satisfy its own postcondition while leaving the robot, objects, or camera views in a state from which the next skill cannot reliably start. We study this semantic handoff problem in BEHAVIOR-1K through an agent-orchestrated vision-language-action execution harness. The harness invokes $π_{0.5}$-based skill checkpoints trained from cleaned BEHAVIOR-1K demonstrations, assigns each skill typed arguments and a step budget, and uses multi-view vision-language model verification to decide whether execution should advance, retry, or replan. To separate isolated skill competence from long-horizon compositional robustness, we evaluate the same checkpoints under two initial-state distributions: clean skill-boundary snapshots and chained terminal states produced by previous skills. Selected navigation, grasping, placement, and door-opening skills achieve 77--100% success from clean snapshots under human-reviewed verification, yet composed rollouts still frequently stall from chained states. The resulting traces attribute failures to next-skill readiness, target grounding, and control execution, turning nearzero task success into actionable diagnostics for what VLA skill libraries must learn next: robustness to the messy chained-state distribution that clean demonstrations underrepresent.

Related papers

• Behavior-Skill: A Fine-Grained Benchmark for Evaluating Vision-Language-Action Policies in Long-Horizon Tasks [39.959375504916494]
  We present Behavior-Skill, a benchmark that reformulates the learning and evaluation of long-horizon tasks.<n>It contains 235,492 skill instances from 10,000 demonstrations across 50 household tasks and 34 semantic skill categories.<n>We show that Behavior-Skill complements full-task evaluation by exposing intermediate capability profiles for analyzing and improving long-horizon VLA policies.
  arXiv  Detail & Related papers  (2026-08-31T10:03:48Z)
• Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills [53.15216425100295]
  We introduce Skill Self-Play (Skill-SP), a co-evolutionary framework comprising a proposer, a solver, and a skill controller.<n>The proposer generates challenging tasks conditioned on dynamically sampled skills; the solver explores candidate solutions to push its capability boundaries; and the skill controller collects execution feedback to update and expand the skill library.<n>This interactive co-evolution effectively bridges the gap between structured verification and open-ended exploration.
  arXiv  Detail & Related papers  (2026-07-24T17:59:22Z)
• SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use [21.05387636058484]
  SkillCoach is a self-evolving framework for evaluating and enhancing agentic skill-use.<n>It derives skill-grounded process rubrics from real rollouts and evaluates trajectories along four dimensions.<n>It keeps the external verifier as a separate outcome signal, allowing process quality to be distinguished from accidental task success.
  arXiv  Detail & Related papers  (2026-07-02T08:28:51Z)
• SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment [10.850946672552277]
  We introduce SkillAudit, an end-to-end framework for skill-centered assessment.<n>It generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety.<n>After scanning top-ranked real-world skill packages, we found that over 7% of skills are at risky status.
  arXiv  Detail & Related papers  (2026-06-21T17:40:05Z)
• Learning New Tasks via Reusable Skills: Skill-Compositional Experts for Embodied Continual Learning [60.37693496953694]
  Embodied Continual Learning (ECL) aims to enable robots to continually acquire new manipulation tasks.<n>ECL suffers from more severe catastrophic forgetting.<n>Key challenge in ECL lies in structured skill reuse across continually evolving tasks.
  arXiv  Detail & Related papers  (2026-06-14T09:02:49Z)
• VASO: Formally Verifiable Self-Evolving Skills for Physical AI Agents [57.240036084348354]
  We introduce VASO, a framework for verification-guided self-evolution of robot skill contracts.<n>VASO verifies logically inconsistent skill contracts, then verifies plans induced by the skill against global and local temporal specifications.<n>On Clearpath Jackal and PX4 quadcopter tasks, VASO reaches 97.2% formal-specification compliance using fewer than 100 optimization samples.
  arXiv  Detail & Related papers  (2026-06-03T20:02:35Z)
• SkillGen: Verified Inference-Time Agent Skill Synthesis [60.927977774369516]
  SkillGen is a framework that synthesizes a single auditable skill from trajectories generated by a base agent.<n>It identifies reusable success patterns, recurring failure modes, and behaviors that appear in nearby successes but are missing from failures.<n>A key novelty in SkillGen is that we model agent skills as interventions to empirically verify the net effect of skills on the overall performance.
  arXiv  Detail & Related papers  (2026-05-09T19:24:11Z)
• Offline Discovery of Interpretable Skills from Multi-Task Trajectories [8.119611773942562]
  We introduce LOKI, a three-stage end-to-end learning framework for offline skill discovery and hierarchical imitation.<n>LOKI achieves high success rates on the challenging D4RL Kitchen benchmark and outperforms standard HIL baselines.
  arXiv  Detail & Related papers  (2026-02-01T05:03:58Z)
• Hierarchical Vision Language Action Model Using Success and Failure Demonstrations [60.82332413442677]
  We introduce VINE, a hierarchical vision-language-action model that separates high-level reasoning from low-level control.<n>System 2 performs feasibility-guided tree search over a 2D scene-graph abstraction.<n>System 1 executes low-level actions without modifying the agent's core skills.
  arXiv  Detail & Related papers  (2025-12-03T15:58:38Z)
• SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution [75.2573501625811]
  Diffusion models have demonstrated strong potential for robotic trajectory planning. generating coherent trajectories from high-level instructions remains challenging. We propose SkillDiffuser, an end-to-end hierarchical planning framework.
  arXiv  Detail & Related papers  (2023-12-18T18:16:52Z)
• Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks [85.56153200251713]
  We introduce EMBR, a model-based RL method for learning primitive skills that are suitable for completing long-horizon visuomotor tasks. On a Franka Emika robot arm, we find that EMBR enables the robot to complete three long-horizon visuomotor tasks at 85% success rate.
  arXiv  Detail & Related papers  (2021-09-21T16:48:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.