Semantically Structured Mixture-of-Experts for Compositional Robotic Manipulation

• URL: http://arxiv.org/abs/2605.23477v1
• Date: Fri, 22 May 2026 10:38:59 GMT
• Title: Semantically Structured Mixture-of-Experts for Compositional Robotic Manipulation
• Abstract summary: Diffusion-based policies have established a new standard for precise robotic manipulation but face a critical scalability bottleneck.<n>We introduce Semantically Structured Mixture-of-Experts Diffusion Policy (SMoDP) for compositional robotic manipulation.<n>Our approach outperforms representative diffusion and MoE-based baselines on multi-task benchmarks with significantly improved parameter efficiency.
• Score: 21.358519910897652
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion-based policies have established a new standard for precise robotic manipulation but face a critical scalability bottleneck: high-performance models are computationally expensive, while lightweight alternatives often fail to generalize across diverse multi-task environments. Mixture-of-Experts (MoE) architectures offer a promising path to efficiency by activating only a subset of parameters. However, existing MoE routing mechanisms typically rely on low-level noise or latent statistics, ignoring the compositional nature of manipulation tasks. This can fragment reusable behaviors across experts, limiting interpretability and transferability. We introduce Semantically Structured Mixture-of-Experts Diffusion Policy (SMoDP) for compositional robotic manipulation, a framework that grounds expert specialization in semantic task structure. SMoDP leverages a lightweight, inference-time skill predictor, supervised by offline annotations from Vision-Language Models (VLMs), to route action chunks to experts specialized for specific behavioral phases. To ensure robust assignment, we propose a dual contrastive alignment strategy that grounds multi-modal observations in language-defined skill semantics (Inter-modal) while enforcing routing consistency across visually distinct but functionally related behaviors (Intra-modal). Our approach outperforms representative diffusion and MoE-based baselines on multi-task benchmarks with significantly improved parameter efficiency and demonstrates effective compositional transfer to novel tasks through parameter-efficient fine-tuning. Project website: https://deng-cy20.github.io/SMoDP/

Related papers

• ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning [16.21619173438734]
  Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities.<n>Recent methods often employ sparse architectures like Mixture of LoRA Experts with image-text similarity routing.<n>We propose ProtoAda, a prototype-guided adaptive tuning framework.
  arXiv  Detail & Related papers  (2026-06-01T17:59:13Z)
• Learning Multi-Modal Trajectory Policies for Data-Efficient Robotic Manipulation [21.99077052779724]
  MATE (Multi-Modal MoE) is a novel trajectory prediction framework built upon MoE.<n>We introduce a Multi-Modal MoE architecture to achieve fine-grained sub-modal cosine, and design a cross-modal router for stable and scale-invariant expert assignment.<n>Experiments on robotic ping-pong show that our MATE consistently outperforms prior work under data scarcity.
  arXiv  Detail & Related papers  (2026-05-31T06:35:10Z)
• Differentiable Mixture-of-Agents Incentivizes Swarm Intelligence of Large Language Models [17.7409616106227]
  Differentiable Mixture-of-Agents (DMoA) is a self-evolving multi-agent framework that enables elastic and adaptive agent collaboration during inference.<n>We show that DMoA achieves state-of-the-art performance while exhibiting strong efficiency, robustness, and ensembling capabilities.
  arXiv  Detail & Related papers  (2026-05-15T07:54:46Z)
• Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuning [58.90602461025722]
  Multi-task instruct-tuning in large language models (LLMs) has been largely driven by multi-task instruct-tuning.<n>The training paradigm suffers from a key issue, named cross-task interference, due to conflicting gradients over shared parameters among different tasks.<n>We propose BADIT that decomposes LLM parameters into high-singular-value LoRA experts representing basic abilities.<n>We conduct extensive experiments on the SuperNI benchmark with 6 LLMs, and empirical results demonstrate that BADIT can outperform SOTA methods and mitigate the degree of cross-task interference.
  arXiv  Detail & Related papers  (2026-05-07T05:08:58Z)
• Parameter-Efficient Modality-Balanced Symmetric Fusion for Multimodal Remote Sensing Semantic Segmentation [8.840077295284393]
  MoBaNet is a parameter-efficient and modality-balanced symmetric fusion framework.<n>Built upon a largely frozen VFM backbone, MoBaNet adopts a symmetric dual-stream architecture to preserve generalizable representations.<n>Experiments on the ISPRS Vaihingen and Potsdam benchmarks demonstrate that MoBaNet achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-03-18T13:23:58Z)
• Efficient and Interpretable Multi-Agent LLM Routing via Ant Colony Optimization [58.59491516762626]
  We propose AMRO-S, an efficient and interpretable routing framework for Multi-Agent Systems (MAS)<n>AMRO-S models MAS routing as a semantic-conditioned path selection problem, enhancing routing performance through three key mechanisms.<n>Extensive experiments on five public benchmarks and high-concurrency stress tests demonstrate that AMRO-S consistently improves the quality--cost trade-off over strong routing baselines.
  arXiv  Detail & Related papers  (2026-03-13T12:26:05Z)
• Expert Pyramid Tuning: Efficient Parameter Fine-Tuning for Expertise-Driven Task Allocation [7.579163774245479]
  Expert Pyramid Tuning (EPT) is a novel architecture that integrates the multi-scale feature pyramid concept from computer vision into the realm of PEFT.<n>EPT significantly outperforms SOTA MoE-LoRA variants.
  arXiv  Detail & Related papers  (2026-03-13T02:23:41Z)
• Parameter Aware Mamba Model for Multi-task Dense Prediction [69.94454603308196]
  We introduce a novel decoder-based framework, Aware Mamba Model (PAMM), specifically designed for dense prediction in multi-task learning setting.<n>It features dual state space parameter experts that integrate and set task-specific parameter priors, capturing the intrinsic properties of each task.<n>We employ the Multi-Directional Hilbert Scanning method to construct multi-angle feature sequences, thereby enhancing the sequence model's perceptual capabilities for 2D data.
  arXiv  Detail & Related papers  (2025-11-18T13:48:00Z)
• Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidance [79.21541758879012]
  Mixture-of-Experts (MoE) has emerged as a powerful paradigm for scaling model capacity while preserving computational efficiency.<n>We present ProMoE, an MoE framework featuring a two-step router with explicit routing guidance that promotes expert specialization.
  arXiv  Detail & Related papers  (2025-10-28T17:59:02Z)
• ContextNav: Towards Agentic Multimodal In-Context Learning [85.05420047017513]
  ContextNav is an agentic framework that integrates the scalability of automated retrieval with the quality and adaptiveness of human-like curation.<n>It builds a resource-aware multimodal embedding pipeline, maintains a retrievable vector database, and applies agentic retrieval and structural alignment to construct noise-resilient contexts.<n> Experimental results demonstrate that ContextNav achieves state-of-the-art performance across various datasets.
  arXiv  Detail & Related papers  (2025-10-06T07:49:52Z)
• Towards Agentic AI for Multimodal-Guided Video Object Segmentation [14.877182670778284]
  Referring-based Video Object is a multimodal problem that requires producing fine-grained segmentation results guided by external cues.<n>Recent advances in vision-language foundation models open a promising direction toward training-free approaches.<n>We propose Multi-Modal Agent, a novel agentic system designed to solve this task in a more flexible and adaptive manner.
  arXiv  Detail & Related papers  (2025-08-14T12:11:15Z)
• T-REX: Mixture-of-Rank-One-Experts with Semantic-aware Intuition for Multi-task Large Language Model Finetuning [31.276142111455847]
  Large language models (LLMs) encounter significant adaptation challenges in diverse multitask finetuning.<n>We design a novel framework, mixunderlinetextbfTureunderlinetextbf-of-underlinetextbfRank-onunderlinetextbfE-eunderlinetextbfXper ts (textttT-REX)<n>Rank-1 experts enable a mix-and-match mechanism to quadratically expand the vector subspace of experts with linear parameter overheads, achieving approximate error reduction with optimal
  arXiv  Detail & Related papers  (2024-04-13T12:14:58Z)
• InterroGate: Learning to Share, Specialize, and Prune Representations for Multi-task Learning [17.66308231838553]
  We propose a novel multi-task learning (MTL) architecture designed to mitigate task interference while optimizing inference computational efficiency. We employ a learnable gating mechanism to automatically balance the shared and task-specific representations while preserving the performance of all tasks.
  arXiv  Detail & Related papers  (2024-02-26T18:59:52Z)
• Learning Robust State Abstractions for Hidden-Parameter Block MDPs [55.31018404591743]
  We leverage ideas of common structure from the HiP-MDP setting to enable robust state abstractions inspired by Block MDPs. We derive instantiations of this new framework for both multi-task reinforcement learning (MTRL) and meta-reinforcement learning (Meta-RL) settings.
  arXiv  Detail & Related papers  (2020-07-14T17:25:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.