Language-Critique Imitation Learning from Suboptimal Demonstrations

• URL: http://arxiv.org/abs/2607.01225v1
• Date: Wed, 01 Jul 2026 17:57:22 GMT
• Title: Language-Critique Imitation Learning from Suboptimal Demonstrations
• Abstract summary: We propose a language-critique framework for imitation learning from suboptimal demonstrations.<n>Our method first constructs language labels from demonstrations that explicitly describe current progress.<n>We then introduce a language-critique loss that directly trains policies using these structured signals.
• Score: 23.463660165952632
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prior work on imitation learning from suboptimal demonstrations typically relies on compressed supervision signals such as confidence estimates, discriminator scores, or importance weights. These scalar signals are inherently limited, as they cannot explicitly express intermediate reasoning about task progress, failure modes, or corrective actions. We propose a language-critique framework for imitation learning from suboptimal demonstrations that instead leverages natural language as a structured supervision signal, avoiding the collapse of expressive feedback into scalars. Our method first constructs language labels from demonstrations that explicitly describe current progress, identify suboptimal behaviors, and provide fine-grained corrective guidance. We then introduce a language-critique loss that directly trains policies using these structured signals without reducing them to scalars, and instantiate it for both behavior cloning and diffusion policies, yielding LC-BC and LC-DP. We further provide a theoretical result showing that the proposed objective upper-bounds the expert performance gap under standard assumptions. Empirically, we evaluate on diverse continuous control tasks spanning navigation, manipulation, and gameplay, where our methods consistently outperform strong imitation learning and offline reinforcement learning baselines. These results demonstrate that language can serve as a powerful and structured form of supervision for learning robust policies from suboptimal data.

Related papers

• In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use [18.507118969955666]
  Vision-Language-Action models imitate expert action chunks conditioned on a static image and a fixed instruction.<n>We introduce textbfourmethod, a framework that endows a VLA with language competence through in-context post-training.<n>Our method consistently achieves SOTA results in both performance and efficiency when compared with CoT-based approaches.
  arXiv  Detail & Related papers  (2026-08-06T08:21:02Z)
• STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning [80.78140312980484]
  We propose language-driven stepwise trajectory redirection, termed as STRIDE.<n>We co-train a generator and a generative verifier using only outcome-based rewards, eliminating external annotations.<n> Experiments on diverse reasoning benchmarks show that STRIDE significantly outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-05-13T11:04:31Z)
• Robust Skills, Brittle Grounding: Diagnosing Restricted Generalization in Vision-Language Action Policies via Multi-Object Picking [6.713814849613683]
  Vision-language action (VLA) policies often report strong manipulation benchmark performance with relatively few demonstrations.<n>We present a controlled multi-object picking study that progressively increases object placement variability up to full workspace randomization.<n>We find that for representative VLA policies, including SmolVLA and $_0.5$, execution of the manipulation primitive remains substantially more reliable than instruction-conditioned task success.
  arXiv  Detail & Related papers  (2026-02-27T16:20:04Z)
• Stable Language Guidance for Vision-Language-Action Models [62.80963701282789]
  Residual Semantic Steering is a probabilistic framework that disentangles physical affordance from semantic execution.<n> RSS achieves state-of-the-art robustness, maintaining performance even under adversarial linguistic perturbations.
  arXiv  Detail & Related papers  (2026-01-07T16:16:10Z)
• DAIL: Beyond Task Ambiguity for Language-Conditioned Reinforcement Learning [28.027785116421242]
  We present DAIL (Distributional Aligned Learning), featuring two key components: distributional policy and semantic alignment.<n>We show that DAIL effectively resolves instruction ambiguities, achieving superior performance to baseline methods.
  arXiv  Detail & Related papers  (2025-10-22T13:16:46Z)
• FOSSIL: Harnessing Feedback on Suboptimal Samples for Data-Efficient Generalisation with Imitation Learning for Embodied Vision-and-Language Tasks [45.65159253753118]
  This work explores how agents trained with imitation learning can learn robust representations from both optimal and suboptimal demonstrations.<n>We provide language feedback embeddings as part of the input sequence into a Transformer-based policy.<n>We test our approach on a range of embodied Vision-and-Language tasks in our custom BabyAI-XGen environment.
  arXiv  Detail & Related papers  (2025-10-13T11:55:21Z)
• Learning a Dense Reasoning Reward Model from Expert Demonstration via Inverse Reinforcement Learning [50.20267980386502]
  We learn a dense, token-level reward model for process supervision directly from expert demonstrations.<n>The learned reasoning reward serves two complementary roles: (i) it provides step-level feedback to optimise a reasoning policy during training; and (ii) it functions at inference as a critic to rerank sampled traces under fixed compute budgets.
  arXiv  Detail & Related papers  (2025-10-02T09:55:26Z)
• Do What? Teaching Vision-Language-Action Models to Reject the Impossible [53.40183895299108]
  Vision-Language-Action (VLA) models have demonstrated strong performance on a range of robotic tasks.<n>We propose Instruct-Verify-and-Act (IVA), a framework that detects when an instruction cannot be executed due to a false premise.<n>Our experiments show that IVA improves false premise detection accuracy by 97.56% over baselines.
  arXiv  Detail & Related papers  (2025-08-22T10:54:33Z)
• Disentangling Latent Shifts of In-Context Learning with Weak Supervision [0.0]
  In-context learning (ICL) enables large language models to perform few-shot learning by conditioning on labeled examples in the prompt.<n>Despite its flexibility, ICL suffers from instability as prompt length increases with more demonstrations.<n>We propose a parameter-efficient method that disentangles demonstration-induced latent shifts from those of the query.
  arXiv  Detail & Related papers  (2024-10-02T13:00:21Z)
• Sentence Representation Learning with Generative Objective rather than Contrastive Objective [86.01683892956144]
  We propose a novel generative self-supervised learning objective based on phrase reconstruction. Our generative learning achieves powerful enough performance improvement and outperforms the current state-of-the-art contrastive methods.
  arXiv  Detail & Related papers  (2022-10-16T07:47:46Z)
• Improving Policy Learning via Language Dynamics Distillation [87.27583619910338]
  We propose Language Dynamics Distillation (LDD), which pretrains a model to predict environment dynamics given demonstrations with language descriptions. We show that language descriptions in demonstrations improve sample-efficiency and generalization across environments.
  arXiv  Detail & Related papers  (2022-09-30T19:56:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.