Behavior-Skill: A Fine-Grained Benchmark for Evaluating Vision-Language-Action Policies in Long-Horizon Tasks

• URL: http://arxiv.org/abs/2608.30536v1
• Date: Mon, 31 Aug 2026 10:03:48 GMT
• Title: Behavior-Skill: A Fine-Grained Benchmark for Evaluating Vision-Language-Action Policies in Long-Horizon Tasks
• Abstract summary: We present Behavior-Skill, a benchmark that reformulates the learning and evaluation of long-horizon tasks.<n>It contains 235,492 skill instances from 10,000 demonstrations across 50 household tasks and 34 semantic skill categories.<n>We show that Behavior-Skill complements full-task evaluation by exposing intermediate capability profiles for analyzing and improving long-horizon VLA policies.
• Score: 39.959375504916494
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reliable execution of long-horizon mobile manipulation tasks remains challenging because overall task success depends on the successful completion of multiple constituent skills. Existing benchmarks, however, still rely primarily on full-task rollouts and aggregate task-level metrics, making intermediate failures difficult to observe and analyze. We present Behavior-Skill, a benchmark that reformulates the learning and evaluation of long-horizon tasks around executable constituent skills. It contains 235,492 skill instances from 10,000 demonstrations across 50 household tasks and 34 semantic skill categories. Each instance pairs a skill instruction with an aligned observation-action segment, and is further associated with a restorable intermediate state and a skill success condition to enable independent evaluation under valid preconditions. We further introduce trajectory-level and skill-level metrics to characterize policy capability beyond aggregate task success. Extensive experiments across representative VLA policies including pi0.5 and GR00T on the complete 50-task benchmark show that failures are highly non-uniform across skills, with contact-rich manipulation skills forming persistent bottlenecks. These results demonstrate that Behavior-Skill complements full-task evaluation by exposing intermediate capability profiles for analyzing and improving long-horizon VLA policies. Behavior-Skill is publicly available at https://github.com/nubot-nudt/Behavior-Skill.

Related papers

• EmbodiedSkills: A Unified Framework for Orchestrating, Training, and Deploying VLA Agents [56.25120535601186]
  Vision-language-action (VLA) models map visual observations and language instructions directly to robot actions.<n>We propose EmbodiedSkills, a framework that treats each skill decision as an execution proposal.<n>A shared executable-skill interface connects high-level skill selection, bounded low-level VLA execution, and post-action verification.
  arXiv  Detail & Related papers  (2026-09-01T14:14:47Z)
• Hierarchical Skill Retrieval for Data-Efficient Adaptation of Vision-Language-Action Models [10.94594187411586]
  Hierarchical Skill Retrieval (HSR) is a retrieval framework for data-efficient VLA adaptation.<n>We show HSR improves the average success rate by 10.3% and 21.3% over the strongest baseline.
  arXiv  Detail & Related papers  (2026-08-25T04:04:49Z)
• SkillRise: Agentic Reinforcement Learning for Cross-Task Skill Evolution [39.74364990208475]
  Large language model agents often encounter related yet distinct tasks that share reusable solution patterns.<n>We introduce SkillRise, a unified reinforcement learning framework for learning skills across tasks.
  arXiv  Detail & Related papers  (2026-07-29T11:26:33Z)
• Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills [53.15216425100295]
  We introduce Skill Self-Play (Skill-SP), a co-evolutionary framework comprising a proposer, a solver, and a skill controller.<n>The proposer generates challenging tasks conditioned on dynamically sampled skills; the solver explores candidate solutions to push its capability boundaries; and the skill controller collects execution feedback to update and expand the skill library.<n>This interactive co-evolution effectively bridges the gap between structured verification and open-ended exploration.
  arXiv  Detail & Related papers  (2026-07-24T17:59:22Z)
• Diagnosing Semantic Handoff Failures in Agent-Orchestrated Vision-Language-Action Skill Composition [10.276503485743111]
  Long-horizon household tasks require robots to compose many language-conditioned skills.<n>A skill may satisfy its own postcondition while leaving the robot in a state from which the next skill cannot reliably start.<n>We study this semantic handoff problem in BEHAVIOR-1K through an agent-orchestrated vision-language-action execution harness.
  arXiv  Detail & Related papers  (2026-07-07T13:24:37Z)
• DexJoCo: A Benchmark and Toolkit for Task-Oriented Dexterous Manipulation on MuJoCo [82.73355932907565]
  DexJoCo is a benchmark and toolkit for task-oriented dexterous manipulation.<n>We present 11 functionally grounded tasks that evaluate tool-use, bimanual coordination, long-horizon execution, and reasoning.<n>We benchmark modern models under diverse settings, including visual and dynamics randomization, multi-task training, and action-head adaptation.
  arXiv  Detail & Related papers  (2026-05-15T17:59:51Z)
• Evidence Over Plans: Online Trajectory Verification for Skill Distillation [59.17625804241461]
  Posterior Distillation Index (PDI) is a trajectory-level metric that quantifies how well a distilled skill is grounded in the task-environment evidence.<n> SPARK generates environment-verified trajectories used to compute PDI.<n>We show that SPARK-generated skills consistently surpass no-skill baselines and outperform human-written skills on student models.
  arXiv  Detail & Related papers  (2026-05-09T22:15:13Z)
• SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents [33.117459452689964]
  We introduce SkillFlow, a benchmark of 166 tasks across 20 families in which task construction within each family follows a Domain-Agnostic Execution Flow (DAEF)<n>Agents are evaluated under an Agentic Lifelong Learning protocol in which they begin without skills, solve tasks sequentially within each family, externalize lessons through trajectory- and rubric-driven skill patches, and carry the updated library forward.
  arXiv  Detail & Related papers  (2026-04-19T07:51:46Z)
• LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces [65.11019654023978]
  LongCLI-Bench is a benchmark designed to evaluate agentic capabilities across long-horizon, realistic tasks.<n>We curated 20 high-quality, long-horizon tasks from over 1,000 computer science assignments and real-world tasks.<n>Experiments reveal that even state-of-the-art agents achieve pass rates below 20% in LongCLI-Bench.
  arXiv  Detail & Related papers  (2026-02-15T23:12:57Z)
• EVOLVE-VLA: Test-Time Training from Environment Feedback for Vision-Language-Action Models [57.75717492488268]
  Vision-Language-Action (VLA) models have advanced robotic manipulation by leveraging large language models.<n>Supervised Finetuning (SFT) requires hundreds of demonstrations per task, rigidly memorizing trajectories, and failing to adapt when deployment conditions deviate from training.<n>We introduce EVOLVE-VLA, a test-time training framework enabling VLAs to continuously adapt through environment interaction with minimal or zero task-specific demonstrations.
  arXiv  Detail & Related papers  (2025-12-16T18:26:38Z)
• STAP: Sequencing Task-Agnostic Policies [22.25415946972336]
  We present Sequencing Task-Agnostic Policies (STAP) for training manipulation skills and coordinating their geometric dependencies at planning time to solve long-horizon tasks. Our experiments indicate that this objective function approximates ground truth plan feasibility. We demonstrate how STAP can be used for task and motion planning by estimating the geometric feasibility of skill sequences provided by a task planner.
  arXiv  Detail & Related papers  (2022-10-21T21:09:37Z)
• Design of Negative Sampling Strategies for Distantly Supervised Skill Extraction [19.43668931500507]
  We propose an end-to-end system for skill extraction, based on distant supervision through literal matching. We observe that using the ESCO taxonomy to select negative examples from related skills yields the biggest improvements. We release the benchmark dataset for research purposes to stimulate further research on the task.
  arXiv  Detail & Related papers  (2022-09-13T13:37:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.