Auditing Instruction-Trajectory Mismatches in Multimodal Robot Demonstrations
- URL: http://arxiv.org/abs/2608.07895v1
- Date: Sat, 08 Aug 2026 03:43:58 GMT
- Title: Auditing Instruction-Trajectory Mismatches in Multimodal Robot Demonstrations
- Abstract summary: We study post-hoc auditing of Instruction-Trajectory Mismatches (ITMs)<n>We propose Multimodal Probabilistic Fusion (MMPF) that treats each modality as an expert, estimates a task-label distribution from local neighborhood agreement and global prototype similarity, and then fuses modalities with predictive-entropy weighting in a product of experts.<n>We demonstrate in real robot experiments that our method can achieve improved policy performance and show the trade-off of filtering demonstrations compared to relabeling.
- Score: 39.205808642879084
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Robot demonstration datasets used to train vision-language-action policies can contain a subtle but harmful failure mode: trajectories that are behaviorally correct but paired with the wrong language instruction. We study post-hoc auditing of these Instruction-Trajectory Mismatches (ITMs). Unlike failed rollouts, ITMs often look plausible, and can corrupt the language-behavior mapping learned by the policy. We propose Multimodal Probabilistic Fusion (MMPF), a training-free auditing framework that treats each modality as an expert, estimates a task-label distribution from local neighborhood agreement and global prototype similarity, and then fuses modalities with predictive-entropy weighting in a product of experts. Across LIBERO benchmarks with injected instruction mismatches and noisy real-robot data, MMPF achieves the strongest overall ITM detection and label correction accuracy. We also show that auditing improves most downstream policy learning in settings where language is needed to disambiguate the task. We demonstrate in real robot experiments that our method can achieve improved policy performance and show the trade-off of filtering demonstrations compared to relabeling.
Related papers
- Learning Multi-Modal Trajectory Policies for Data-Efficient Robotic Manipulation [21.99077052779724]
MATE (Multi-Modal MoE) is a novel trajectory prediction framework built upon MoE.<n>We introduce a Multi-Modal MoE architecture to achieve fine-grained sub-modal cosine, and design a cross-modal router for stable and scale-invariant expert assignment.<n>Experiments on robotic ping-pong show that our MATE consistently outperforms prior work under data scarcity.
arXiv Detail & Related papers (2026-05-31T06:35:10Z) - WM-DAgger: Enabling Efficient Data Aggregation for Imitation Learning with World Models [22.67666648771421]
We propose WM-DAgger, an efficient data aggregation framework.<n>We focus on manipulation tasks with an eye-in-hand robotic arm and only few-shot demonstrations.<n>We extensively validate WM-DAgger on multiple real-world robotic tasks.
arXiv Detail & Related papers (2026-04-13T11:49:11Z) - DetectAnyLLM: Towards Generalizable and Robust Detection of Machine-Generated Text Across Domains and Models [60.713908578319256]
We propose Direct Discrepancy Learning (DDL) to optimize the detector with task-oriented knowledge.<n>Built upon this, we introduce DetectAnyLLM, a unified detection framework that achieves state-of-the-art MGTD performance.<n>MIRAGE samples human-written texts from 10 corpora across 5 text-domains, which are then re-generated or revised using 17 cutting-edge LLMs.
arXiv Detail & Related papers (2025-09-15T10:59:57Z) - CUPID: Curating Data your Robot Loves with Influence Functions [29.79350259314518]
CUPID is a robot data curation method based on a novel influence function-theoretic formulation for imitation learning policies.<n>We use CUPID to curate data by 1) filtering out training demonstrations that harm policy performance and 2) subselecting newly collected trajectories that will most improve the policy.
arXiv Detail & Related papers (2025-06-23T20:49:34Z) - Curating Demonstrations using Online Experience [52.59275477573012]
We show that Demo-SCORE can effectively identify suboptimal demonstrations without manual curation.<n>Demo-SCORE achieves over 15-35% higher absolute success rate in the resulting policy compared to the base policy trained with all original demonstrations.
arXiv Detail & Related papers (2025-03-05T17:58:16Z) - SPARC: Score Prompting and Adaptive Fusion for Zero-Shot Multi-Label Recognition in Vision-Language Models [74.40683913645731]
Zero-shot multi-label recognition (MLR) with Vision-Language Models (VLMs) faces significant challenges without training data, model tuning, or architectural modifications.<n>Our work proposes a novel solution treating VLMs as black boxes, leveraging scores without training data or ground truth.<n>Analysis of these prompt scores reveals VLM biases and AND''/OR' signal ambiguities, notably that maximum scores are surprisingly suboptimal compared to second-highest scores.
arXiv Detail & Related papers (2025-02-24T07:15:05Z) - Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method [108.56493934296687]
We introduce a divergence-based calibration method, inspired by the divergence-from-randomness concept, to calibrate token probabilities for pretraining data detection.<n>We have developed a Chinese-language benchmark, PatentMIA, to assess the performance of detection approaches for LLMs on Chinese text.
arXiv Detail & Related papers (2024-09-23T07:55:35Z) - Aligning Language Models with Demonstrated Feedback [58.834937450242975]
Demonstration ITerated Task Optimization (DITTO) directly aligns language model outputs to a user's demonstrated behaviors.<n>We evaluate DITTO's ability to learn fine-grained style and task alignment across domains such as news articles, emails, and blog posts.
arXiv Detail & Related papers (2024-06-02T23:13:56Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.