Skill-Aware Diffusion for Generalizable Robotic Manipulation

• URL: http://arxiv.org/abs/2601.11266v1
• Date: Fri, 16 Jan 2026 13:14:40 GMT
• Title: Skill-Aware Diffusion for Generalizable Robotic Manipulation
• Abstract summary: We propose Skill-Aware Diffusion (SADiff), which explicitly incorporates skill-level information to improve generalization.<n>SADiff learns skill-specific representations through a skill-aware encoding module with learnable skill tokens.<n> Experiments in simulation and real-world settings show that SADiff achieves good performance and generalization across various manipulation tasks.
• Score: 10.771128286526634
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robust generalization in robotic manipulation is crucial for robots to adapt flexibly to diverse environments. Existing methods usually improve generalization by scaling data and networks, but model tasks independently and overlook skill-level information. Observing that tasks within the same skill share similar motion patterns, we propose Skill-Aware Diffusion (SADiff), which explicitly incorporates skill-level information to improve generalization. SADiff learns skill-specific representations through a skill-aware encoding module with learnable skill tokens, and conditions a skill-constrained diffusion model to generate object-centric motion flow. A skill-retrieval transformation strategy further exploits skill-specific trajectory priors to refine the mapping from 2D motion flow to executable 3D actions. Furthermore, we introduce IsaacSkill, a high-fidelity dataset containing fundamental robotic skills for comprehensive evaluation and sim-to-real transfer. Experiments in simulation and real-world settings show that SADiff achieves good performance and generalization across various manipulation tasks. Code, data, and videos are available at https://sites.google.com/view/sa-diff.

Related papers

• Decoupling the Declarative from the Procedural in Vision-Language-Action Models [65.22639791239932]
  Policy trained to clone a behavior from object-specific demonstrations must generalize beyond that object.<n>We propose w$2$VLA, a new Vision-Language-Action model with restructured information flow.<n>Unlike popular, state-of-the-art VLAs, we show that our modular approach successfully decouples knowledge representations.
  arXiv  Detail & Related papers  (2026-06-19T14:43:58Z)
• Learning Skills from Action-Free Videos [20.442392109789772]
  We introduce Skill Abstraction from Optical Flow (SOF), a framework that learns latent skills from action-free videos.<n>SOF enables high-level planning over video-derived skills and allows for easier translation of these skills into actions.<n> Experiments show that our approach consistently improves performance in both multitask and long-horizon settings.
  arXiv  Detail & Related papers  (2025-12-23T05:03:33Z)
• EL3DD: Extended Latent 3D Diffusion for Language Conditioned Multitask Manipulation [16.468655011980843]
  This paper seeks to harness the capabilities of diffusion models within a visuomotor policy framework to generate precise robotic trajectories.<n>By employing reference demonstrations during training, the model learns to execute manipulation tasks specified through textual commands within the robot's immediate environment.
  arXiv  Detail & Related papers  (2025-11-17T12:47:18Z)
• SPECI: Skill Prompts based Hierarchical Continual Imitation Learning for Robot Manipulation [3.1997825444285457]
  Real-world robot manipulation in dynamic unstructured environments requires lifelong adaptability to evolving objects, scenes and tasks.<n>Traditional imitation learning relies on static training paradigms, which are ill-suited for lifelong adaptation.<n>We propose Skill Prompts-based HiErarchical Continual Imitation Learning (SPECI), a novel end-to-end hierarchical CIL policy architecture for robot manipulation.
  arXiv  Detail & Related papers  (2025-04-22T03:30:38Z)
• SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution [75.2573501625811]
  Diffusion models have demonstrated strong potential for robotic trajectory planning. generating coherent trajectories from high-level instructions remains challenging. We propose SkillDiffuser, an end-to-end hierarchical planning framework.
  arXiv  Detail & Related papers  (2023-12-18T18:16:52Z)
• Human-oriented Representation Learning for Robotic Manipulation [64.59499047836637]
  Humans inherently possess generalizable visual representations that empower them to efficiently explore and interact with the environments in manipulation tasks. We formalize this idea through the lens of human-oriented multi-task fine-tuning on top of pre-trained visual encoders. Our Task Fusion Decoder consistently improves the representation of three state-of-the-art visual encoders for downstream manipulation policy-learning.
  arXiv  Detail & Related papers  (2023-10-04T17:59:38Z)
• RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot [56.130215236125224]
  A key challenge in robotic manipulation in open domains is how to acquire diverse and generalizable skills for robots. Recent research in one-shot imitation learning has shown promise in transferring trained policies to new tasks based on demonstrations. This paper aims to unlock the potential for an agent to generalize to hundreds of real-world skills with multi-modal perception.
  arXiv  Detail & Related papers  (2023-07-02T15:33:31Z)
• ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters [123.88692739360457]
  General-purpose motor skills enable humans to perform complex tasks. These skills also provide powerful priors for guiding their behaviors when learning new tasks. We present a framework for learning versatile and reusable skill embeddings for physically simulated characters.
  arXiv  Detail & Related papers  (2022-05-04T06:13:28Z)
• Hierarchical Few-Shot Imitation with Skill Transition Models [66.81252581083199]
  Few-shot Imitation with Skill Transition Models (FIST) is an algorithm that extracts skills from offline data and utilizes them to generalize to unseen tasks. We show that FIST is capable of generalizing to new tasks and substantially outperforms prior baselines in navigation experiments.
  arXiv  Detail & Related papers  (2021-07-19T15:56:01Z)
• Learning and Sequencing of Object-Centric Manipulation Skills for Industrial Tasks [16.308562047398542]
  We propose a rapid robot skill-sequencing algorithm, where the skills are encoded by object-centric hidden semi-Markov models. The learned skill models can encode multimodal (temporal and spatial) trajectory distributions. We demonstrate this approach on a 7 DoF robot arm for industrial assembly tasks.
  arXiv  Detail & Related papers  (2020-08-24T14:20:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.