SkillMemo: Expert-guided Skill Memory Framework for Compositional Embodied Manipulation

• URL: http://arxiv.org/abs/2608.05970v1
• Date: Thu, 06 Aug 2026 12:46:13 GMT
• Title: SkillMemo: Expert-guided Skill Memory Framework for Compositional Embodied Manipulation
• Abstract summary: Embodied visuomotor models, including Diffusion Policy (DP) and Vision-Language-Action (VLA) models, have demonstrated promising performance on robotic manipulation benchmarks.<n>We propose Skill-Based Memory (SkillMemo) framework that implicitly decomposes long-horizon demonstrations into latent atomic skills.
• Score: 72.14179740056073
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Embodied visuomotor models, including Diffusion Policy (DP) and Vision-Language-Action (VLA) models, have demonstrated promising performance on robotic manipulation benchmarks. However, their potential remains fundamentally constrained by the scarcity of large-scale embodied trajectory datasets, leading to insufficient compositional generalization in out-of-distribution (OOD) scenarios with limited capability to capture reusable skill structures. To address this limitation, we propose Skill-Based Memory (SkillMemo) framework that implicitly decomposes long-horizon demonstrations into latent atomic skills and integrates skill-level features into a dynamic episodic memory bank for solving compositional tasks. Specifically, we first introduce an expert-guided trajectory segmentation module built upon a Mixture-of-Experts (MoE) architecture, which implicitly partitions trajectories into distinct skill primitives represented by learned gating coefficients. We further design a skill-level episodic memory architecture that stores compact skill representations as retrievable key-value pairs. During inference, the memory bank retrieves the most relevant skill primitives which are subsequently fused with the model's current gating distribution, providing a robust contextual prior to refine action predictions. Extensive experiments on the simulation benchmark and real-world manipulation tasks demonstrate that SkillMemo consistently enhances both DP and VLA backbones, achieving state-of-the-art performance and outperforming $π_{0.5}$, while exhibiting strong compositional generalization to unseen task configurations.

Related papers

• Decoupling the Declarative from the Procedural in Vision-Language-Action Models [65.22639791239932]
  Policy trained to clone a behavior from object-specific demonstrations must generalize beyond that object.<n>We propose w$2$VLA, a new Vision-Language-Action model with restructured information flow.<n>Unlike popular, state-of-the-art VLAs, we show that our modular approach successfully decouples knowledge representations.
  arXiv  Detail & Related papers  (2026-06-19T14:43:58Z)
• Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models [50.34585122347149]
  We introduce Expert Tying, an architectural modification that shares expert parameters across consecutive transformer layers.<n>We evaluate this approach across common, state-of-the-art architectures, including OLMoE, Qwen3, and DeepSeek-style MoEs.
  arXiv  Detail & Related papers  (2026-06-15T15:08:09Z)
• Eigenvectors of Experts are Training-free Non-collapsing Routers [14.774596844618396]
  Sparse Mixture of Experts (SMoE) architectures improve the training efficiency of Large Language Models (LLMs) by routing input tokens to a subset of specialized experts.<n>Despite their remarkable success, both training and inference in SMoE models suffer from the expert collapse issue.<n>We propose Singular Value Decomposition SMoE (SSMoE), a novel and training-free framework that leverages spectral properties of the expert weights to address the collapse issue.
  arXiv  Detail & Related papers  (2026-05-29T08:27:10Z)
• CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning [16.033361968793162]
  Catastrophic forgetting remains a major obstacle to continual learning in large language models (LLMs) and vision--language models (VLMs)<n>We propose CP-MoE, a continual learning framework built around a transient expert that captures early task-specific updates and guides their integration into stable experts.<n>We validate CP-MoE on both unimodal and multimodal continual learning benchmarks with LLM-based and VLM-based MoE models.
  arXiv  Detail & Related papers  (2026-05-18T06:31:14Z)
• ECHO: Continuous Hierarchical Memory for Vision-Language-Action Models [40.14860028616163]
  Memory capacity is a critical factor determining the performance of Vision-Language-Action (VLA) models in long-horizon manipulation tasks.<n>We propose ECHO, a novel memory framework operating within a Continuous Hierarchical Space.<n>By employing a hyperbolic autoencoder, ECHO maps VLA hidden states into this space.
  arXiv  Detail & Related papers  (2026-05-09T13:06:33Z)
• $M^2$-VLA: Boosting Vision-Language Models for Generalizable Manipulation via Layer Mixture and Meta-Skills [25.17452377052361]
  Current Vision-Language-Action (VLA) models predominantly rely on end-to-end fine-tuning.<n>We propose $M2$-VLA, which demonstrates that a generalized VLM is able to serve as a powerful backbone for robotic manipulation directly.<n>To overcome this, we introduce the Mixture of Layers (MoL) strategy that selectively extracts task-critical information from dense semantic features.
  arXiv  Detail & Related papers  (2026-04-27T08:44:12Z)
• Going with the Flow: Koopman Behavioral Models as Implicit Planners for Visuo-Motor Dexterity [1.6865201067073452]
  We introduce Unified Behavioral Models (UBMs), a framework that learns to represent dexterous skills as coupled dynamical systems.<n>Koopman-UBM is a first instantiation of UBMs that learns a unified representation in which the joint flow of latent visual and proprioceptive features is governed by a structured linear system.
  arXiv  Detail & Related papers  (2026-02-07T07:18:00Z)
• Foundation Model for Skeleton-Based Human Action Understanding [56.89025287217221]
  This paper presents a Unified Skeleton-based Dense Representation Learning framework.<n>USDRL consists of a Transformer-based Dense Spatio-Temporal (DSTE), Multi-Grained Feature Decorrelation (MG-FD), and Multi-Perspective Consistency Training (MPCT)
  arXiv  Detail & Related papers  (2025-08-18T02:42:16Z)
• Quantifying Memory Utilization with Effective State-Size [73.52115209375343]
  We develop a measure of textitmemory utilization'<n>This metric is tailored to the fundamental class of systems with textitinput-invariant and textitinput-varying linear operators
  arXiv  Detail & Related papers  (2025-04-28T08:12:30Z)
• ContextFormer: Redefining Efficiency in Semantic Segmentation [48.81126061219231]
  Convolutional methods, although capturing local dependencies well, struggle with long-range relationships.<n>Vision Transformers (ViTs) excel in global context capture but are hindered by high computational demands.<n>We propose ContextFormer, a hybrid framework leveraging the strengths of CNNs and ViTs in the bottleneck to balance efficiency, accuracy, and robustness for real-time semantic segmentation.
  arXiv  Detail & Related papers  (2025-01-31T16:11:04Z)
• Skeleton2vec: A Self-supervised Learning Framework with Contextualized Target Representations for Skeleton Sequence [56.092059713922744]
  We show that using high-level contextualized features as prediction targets can achieve superior performance. Specifically, we propose Skeleton2vec, a simple and efficient self-supervised 3D action representation learning framework. Our proposed Skeleton2vec outperforms previous methods and achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2024-01-01T12:08:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.