Hierarchical Skill Retrieval for Data-Efficient Adaptation of Vision-Language-Action Models

• URL: http://arxiv.org/abs/2608.24042v1
• Date: Tue, 25 Aug 2026 04:04:49 GMT
• Title: Hierarchical Skill Retrieval for Data-Efficient Adaptation of Vision-Language-Action Models
• Abstract summary: Hierarchical Skill Retrieval (HSR) is a retrieval framework for data-efficient VLA adaptation.<n>We show HSR improves the average success rate by 10.3% and 21.3% over the strongest baseline.
• Score: 10.94594187411586
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Vision-Language-Action (VLA) models pretrained on large-scale robot datasets provide a strong foundation for robot manipulation, their performance can degrade when adapted to new tasks with limited task-specific demonstrations. Retrieval offers a practical way to reuse existing demonstrations for data-efficient adaptation, but existing methods often rely on visual similarity, state-action representations, or task-level language matching. These approaches may overlook the hierarchical structure of long-horizon manipulation tasks, where complete task matches are rare but reusable skills are often abundant. To address this challenge, we propose Hierarchical Skill Retrieval (HSR), a retrieval framework for data-efficient VLA adaptation. Specifically, HSR first decomposes a target task into candidate skill sequences. It evaluates each plan based on both semantic plausibility and skill reliability estimated from the prior dataset. The selected decomposition is then used for hybrid retrieval. This combines subtask-level language retrieval with behavior-feature reranking to identify demonstrations that are both semantically relevant and compatible with the target task. Finally, we adapt the policy through a two-stage pretraining and finetuning pipeline, which separates general skill acquisition from task-specific adaptation. Experiments on the LIBERO benchmark and several real-world robot manipulation tasks show that HSR improves the average success rate by 10.3% and 21.3% over the strongest baseline, respectively. These results demonstrate the effectiveness of structured skill-level retrieval for data-efficient VLA adaptation. Videos and code are available at https://hoar012.github.io/HSR-Project.

Related papers

• Task-State Adaptation with Prototype Memory for Multi-Task Dense Prediction [65.90697836568579]
  We propose a multi-task dense prediction framework that estimates a compact task state from global visual context.<n>MemMTL refines it through a learnable task-state prototype memory.<n>A separate task-agnostic residual bank provides a common adaptation path.
  arXiv  Detail & Related papers  (2026-08-28T08:48:22Z)
• From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models [73.72877445629383]
  Interpretability-Guided Data Selection (IGDS) is a framework that first identifies causal task features through frequency recall and interventional filtering.<n>We validate IGDS on mathematical reasoning, summarization, and translation tasks within Gemma-2, LLaMA-3.1, and Qwen3 models.
  arXiv  Detail & Related papers  (2026-04-28T03:16:24Z)
• Task-Aware LoRA Adapter Composition via Similarity Retrieval in Vector Databases [3.4869850730657728]
  We present a novel framework for dynamic LoRA adapter composition that leverages similarity retrieval in vector databases.<n>Our approach constructs a task-aware vector database by embedding training examples from 22 spanning commonsense reasoning, question answering, natural language inference, and sentiment analysis.<n>Our framework requires no additional retriever training, operates with frozen embeddings, and enables efficient, interpretable adapter composition.
  arXiv  Detail & Related papers  (2026-02-01T22:20:04Z)
• Vision Language Models are In-Context Value Learners [89.29486557646624]
  We present Generative Value Learning (GVL), a universal value function estimator that leverages the world knowledge embedded in vision-language models (VLMs) to predict task progress. Without any robot or task specific training, GVL can in-context zero-shot and few-shot predict effective values for more than 300 distinct real-world tasks.
  arXiv  Detail & Related papers  (2024-11-07T09:17:50Z)
• P-RAG: Progressive Retrieval Augmented Generation For Planning on Embodied Everyday Task [94.08478298711789]
  Embodied Everyday Task is a popular task in the embodied AI community. Natural language instructions often lack explicit task planning. Extensive training is required to equip models with knowledge of the task environment.
  arXiv  Detail & Related papers  (2024-09-17T15:29:34Z)
• RAVEN: Multitask Retrieval Augmented Vision-Language Learning [5.1583788731239455]
  The scaling of large language models to encode all the world's knowledge is unsustainable and has exacerbated resource barriers. Retrieval-Augmented Generation (RAG) presents a potential solution, yet its application to vision-language models (VLMs) is under explored. This paper introduces RAVEN, a retrieval augmented VLM framework that enhances base VLMs through efficient, task specific fine-tuning.
  arXiv  Detail & Related papers  (2024-06-27T13:08:35Z)
• Adaptive Retention & Correction: Test-Time Training for Continual Learning [114.5656325514408]
  A common problem in continual learning is the classification layer's bias towards the most recent task.<n>We name our approach Adaptive Retention & Correction (ARC)<n>ARC achieves an average performance increase of 2.7% and 2.6% on the CIFAR-100 and Imagenet-R datasets.
  arXiv  Detail & Related papers  (2024-05-23T08:43:09Z)
• Temporal and Semantic Evaluation Metrics for Foundation Models in Post-Hoc Analysis of Robotic Sub-tasks [1.8124328823188356]
  We present an automated framework to decompose trajectory data into temporally bounded and natural language-based descriptive sub-tasks.<n>Our framework provides both time-based and language-based descriptions for lower-level sub-tasks that comprise full trajectories.<n>The metrics measure the temporal alignment and semantic fidelity of language descriptions between two sub-task decompositions.
  arXiv  Detail & Related papers  (2024-03-25T22:39:20Z)
• Task-Distributionally Robust Data-Free Meta-Learning [99.56612787882334]
  Data-Free Meta-Learning (DFML) aims to efficiently learn new tasks by leveraging multiple pre-trained models without requiring their original training data. For the first time, we reveal two major challenges hindering their practical deployments: Task-Distribution Shift ( TDS) and Task-Distribution Corruption (TDC)
  arXiv  Detail & Related papers  (2023-11-23T15:46:54Z)
• Lifelong Learning Without a Task Oracle [13.331659934508764]
  Supervised deep neural networks are known to undergo a sharp decline in the accuracy of older tasks when new tasks are learned. We propose and compare several candidate task-assigning mappers which require very little memory overhead. Best-performing variants only impose an average cost of 1.7% parameter memory increase.
  arXiv  Detail & Related papers  (2020-11-09T21:30:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.