Learning Multi-Modal Trajectory Policies for Data-Efficient Robotic Manipulation

• URL: http://arxiv.org/abs/2606.01047v1
• Date: Sun, 31 May 2026 06:35:10 GMT
• Title: Learning Multi-Modal Trajectory Policies for Data-Efficient Robotic Manipulation
• Abstract summary: MATE (Multi-Modal MoE) is a novel trajectory prediction framework built upon MoE.<n>We introduce a Multi-Modal MoE architecture to achieve fine-grained sub-modal cosine, and design a cross-modal router for stable and scale-invariant expert assignment.<n>Experiments on robotic ping-pong show that our MATE consistently outperforms prior work under data scarcity.
• Score: 21.99077052779724
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robotic manipulation requires the effective integration of heterogeneous inputs, including visual observations, language instructions, and trajectory representations, to generate accurate actions. Existing transformer-based policies typically process these heterogeneous modalities within a shared parameter space, which often leads to modality interference and inefficient representation learning, especially in data-scarce scenarios. While Mixture-of-Experts (MoE) offers a scalable solution through expert specialization, conventional routing mechanisms are often sensitive to such cross-modal representation discrepancies, resulting in unstable expert assignment and expert collapse. In this work, we propose MATE (Multi-ModAl TrajEctory Policies), a novel trajectory prediction framework built upon MoE. Specifically, we introduce a Multi-Modal MoE architecture to achieve fine-grained sub-token feature decoupling, and design a cross-modal cosine router for stable and scale-invariant expert assignment across heterogeneous modalities. We further employ temperature-controlled routing and stochastic noise injection to improve expert balance and prevent premature routing collapse under scarce demonstrations. Experiments on the LIBERO benchmark show that our MATE consistently outperforms prior work under data scarcity. It achieves a 4.75% improvement in average success rate over the trajectory-guided counterpart. Real-world experiments on robotic ping-pong also suggest that the predicted trajectories can provide useful guidance for downstream robotic execution, further indicating the practical feasibility of our algorithm.

Related papers

• Auditing Instruction-Trajectory Mismatches in Multimodal Robot Demonstrations [39.205808642879084]
  We study post-hoc auditing of Instruction-Trajectory Mismatches (ITMs)<n>We propose Multimodal Probabilistic Fusion (MMPF) that treats each modality as an expert, estimates a task-label distribution from local neighborhood agreement and global prototype similarity, and then fuses modalities with predictive-entropy weighting in a product of experts.<n>We demonstrate in real robot experiments that our method can achieve improved policy performance and show the trade-off of filtering demonstrations compared to relabeling.
  arXiv  Detail & Related papers  (2026-08-08T03:43:58Z)
• Model-Based Diffusion Optimal Control for Multi-Robot Motion Planning [5.5818653283217055]
  Multi-Robot Motion Planning in continuous environments is challenging due to the growth of the joint trajectory space and the difficulty of enforcing dynamic feasibility and hard safety constraints.<n>Recent approaches recast trajectory planning as probabilistic inference, sampling from a posterior over trajectories using diffusion models whose score functions are learned from demonstration data.<n>We introduce Model-Based Diffusion Optimal Control (MDOC), a model-based diffusion planner that efficiently produces dynamically feasible trajectories without relying on data.
  arXiv  Detail & Related papers  (2026-07-14T06:54:03Z)
• Semantically Structured Mixture-of-Experts for Compositional Robotic Manipulation [21.358519910897652]
  Diffusion-based policies have established a new standard for precise robotic manipulation but face a critical scalability bottleneck.<n>We introduce Semantically Structured Mixture-of-Experts Diffusion Policy (SMoDP) for compositional robotic manipulation.<n>Our approach outperforms representative diffusion and MoE-based baselines on multi-task benchmarks with significantly improved parameter efficiency.
  arXiv  Detail & Related papers  (2026-05-22T10:38:59Z)
• Adaptor: Advancing Assistive Teleoperation with Few-Shot Learning and Cross-Operator Generalization [26.90003783245582]
  Adaptor is a framework for robust cross-operator intent recognition.<n>It bridges the domain gap through two stages: preprocessing and policy learning.<n>It achieves state-of-the-art performance, improving success rates and efficiency over baselines.
  arXiv  Detail & Related papers  (2026-04-10T16:19:43Z)
• Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons [69.87766750714945]
  General-purpose robot reward models are typically trained to predict absolute task progress from expert demonstrations.<n>We introduce Robometer, a scalable reward modeling framework that combines intra-trajectory progress supervision with inter-trajectory preference supervision.<n>Robometer is trained with a dual objective: a frame-level progress loss that anchors reward magnitude on expert data, and a trajectory-comparison preference loss that imposes global ordering constraints.
  arXiv  Detail & Related papers  (2026-03-02T17:38:58Z)
• Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols [123.73663884421272]
  Few-shot transfer has been revolutionized by stronger pre-trained models and improved adaptation algorithms.<n>We establish FEWTRANS, a comprehensive benchmark containing 10 diverse datasets.<n>By releasing FEWTRANS, we aim to provide a rigorous "ruler" to streamline reproducible advances in few-shot transfer learning research.
  arXiv  Detail & Related papers  (2026-02-28T05:41:57Z)
• Guided Verifier: Collaborative Multimodal Reasoning via Dynamic Process Supervision [11.159231524113764]
  Reinforcement Learning (RL) has emerged as a pivotal mechanism for enhancing the complex reasoning capabilities of Multimodal Large Language Models (MLLMs)<n>In this paper, we propose the textbfGuided Verifier framework to address these structural limitations.<n>We develop a specialized data synthesis pipeline targeting multimodal hallucinations, constructing textbfCoRe dataset of process-level negatives and textbfCorrect-guide textbfReasoning trajectories to train the guided verifier.
  arXiv  Detail & Related papers  (2026-02-04T07:38:42Z)
• Align-Then-stEer: Adapting the Vision-Language Action Models through Unified Latent Guidance [63.33213516925946]
  We introduce textbfAlign-Then-stEer (textttATE), a novel, data-efficient, and plug-and-play adaptation framework.<n>Our work presents a general and lightweight solution that greatly enhances the practicality of deploying VLA models to new robotic platforms and tasks.
  arXiv  Detail & Related papers  (2025-09-02T07:51:59Z)
• Single Domain Generalization with Model-aware Parametric Batch-wise Mixup [22.709796153794507]
  Single Domain Generalization remains a formidable challenge in the field of machine learning.<n>We propose a novel data augmentation approach, named as Model-aware Parametric Batch-wise Mixup.<n>By exploiting inter-feature correlations, the parameterized mixup generator introduces additional versatility in combining features across a batch of instances.
  arXiv  Detail & Related papers  (2025-02-22T03:45:18Z)
• Conditional Neural Expert Processes for Learning Movement Primitives from Demonstration [1.9336815376402723]
  Conditional Neural Expert Processes (CNEP) learns to assign demonstrations from different modes to distinct expert networks. CNEP does not require supervision on which mode the trajectories belong to. Our system is capable of on-the-fly adaptation to environmental changes via an online conditioning mechanism.
  arXiv  Detail & Related papers  (2024-02-13T12:52:02Z)
• Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior [51.60683890503293]
  We propose a theoretical framework for studying behavior cloning of complex expert demonstrations using generative modeling. We show that pure supervised cloning can generate trajectories matching the per-time step distribution of arbitrary expert trajectories.
  arXiv  Detail & Related papers  (2023-07-27T04:27:26Z)
• End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes [52.818579746354665]
  This paper proposes the first end-to-end differentiable meta-BO framework that generalises neural processes to learn acquisition functions via transformer architectures. We enable this end-to-end framework with reinforcement learning (RL) to tackle the lack of labelled acquisition data.
  arXiv  Detail & Related papers  (2023-05-25T10:58:46Z)
• Traj-MAE: Masked Autoencoders for Trajectory Prediction [69.7885837428344]
  Trajectory prediction has been a crucial task in building a reliable autonomous driving system by anticipating possible dangers. We propose an efficient masked autoencoder for trajectory prediction (Traj-MAE) that better represents the complicated behaviors of agents in the driving environment. Our experimental results in both multi-agent and single-agent settings demonstrate that Traj-MAE achieves competitive results with state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-03-12T16:23:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.