When Does Language Matter? Multilingual Instructions Reveal Step-wise Language Sensitivity in Vision-Language-Action Models

• URL: http://arxiv.org/abs/2606.11906v1
• Date: Wed, 10 Jun 2026 10:36:20 GMT
• Title: When Does Language Matter? Multilingual Instructions Reveal Step-wise Language Sensitivity in Vision-Language-Action Models
• Abstract summary: Vision-Language-Action (VLA) models have shown strong performance in language-conditioned robotic manipulation.<n>We present the first systematic multilingual evaluation of VLA models by translating the LIBERO benchmark into ten languages.
• Score: 45.78252494839804
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language-Action (VLA) models have shown strong performance in language-conditioned robotic manipulation, yet their robustness to linguistic variation remains poorly understood. In this work, we present the first systematic multilingual evaluation of VLA models by translating the LIBERO benchmark into ten languages, revealing severe performance degradation under non-English instructions, with success rates dropping by 30-50%. Through fine-grained analysis of task executions, we find that language influence is highly non-uniform across steps: certain steps exhibit strong language dependence and dominate overall task failure, while others are largely language-agnostic. Based on this insight, we propose a step-wise inference-time intervention that aligns representations according to step language sensitivity, substantially improving performance under linguistic variation. Our results indicate that language robustness in VLA models is fundamentally a step-wise control problem, highlighting the importance of temporally structured analysis for reliable embodied agents.

Related papers

• Beyond English: Uncovering the Multilingual Gap in Vision-Language-Action Models [22.61843460495166]
  Vision-Language-Action models have recently demonstrated promising capabilities in learning generalist robot policies from large-scale multimodal data.<n>Most existing VLA systems are trained and evaluated primarily with English instructions, leaving their ability to understand and execute instructions in other languages largely unexplored.<n>We present the first systematic study of multilingual instruction following in VLA models.
  arXiv  Detail & Related papers  (2026-06-14T10:04:08Z)
• LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance [77.58408743830314]
  Reinforcement learning has proven effective for enhancing multi-step reasoning in large language models.<n>But its benefits have not fully translated to multilingual contexts.<n>We develop a novel framework that leverages language-conditioned hints to guide exploration in non-English reasoning tasks.
  arXiv  Detail & Related papers  (2026-05-21T14:47:52Z)
• LangGap: Diagnosing and Closing the Language Gap in Vision-Language-Action Models [4.54067274409672]
  Vision-Language-Action (VLA) models achieve over 95% success on standard benchmarks.<n>We find that current state-of-the-art VLA models largely ignore language instructions.<n>This paper constructs the LangGap benchmark, based on a four-dimensional semantic perturbation method.
  arXiv  Detail & Related papers  (2026-02-28T10:53:33Z)
• Code-Switching In-Context Learning for Cross-Lingual Transfer of Large Language Models [64.54005959758733]
  We introduce code-switching in-context learning (CSICL) as a principled and robust approach for overcoming the translation barrier during inference.<n>We conduct extensive experiments across 4 LLMs, 6 datasets, and 10 languages, spanning both knowledge-intensive and reasoning-oriented domains.<n>Our results demonstrate CSICL consistently outperforms X-ICL baselines, achieving gains of 3.1%p and 1.9%p in both target and unseen languages.
  arXiv  Detail & Related papers  (2025-10-07T08:35:42Z)
• Do What? Teaching Vision-Language-Action Models to Reject the Impossible [53.40183895299108]
  Vision-Language-Action (VLA) models have demonstrated strong performance on a range of robotic tasks.<n>We propose Instruct-Verify-and-Act (IVA), a framework that detects when an instruction cannot be executed due to a false premise.<n>Our experiments show that IVA improves false premise detection accuracy by 97.56% over baselines.
  arXiv  Detail & Related papers  (2025-08-22T10:54:33Z)
• Language Surgery in Multilingual Large Language Models [39.66404344691661]
  Large Language Models (LLMs) have demonstrated remarkable generalization capabilities across tasks and languages.<n>This paper investigates the naturally emerging representation alignment in LLMs, particularly in the middle layers.<n>We propose Inference-Time Language Control (ITLC), a novel method that leverages latent injection to enable precise cross-lingual language control.
  arXiv  Detail & Related papers  (2025-06-14T11:09:50Z)
• Cross-Lingual Pitfalls: Automatic Probing Cross-Lingual Weakness of Multilingual Large Language Models [55.14276067678253]
  This paper introduces a novel methodology for efficiently identifying inherent cross-lingual weaknesses in Large Language Models (LLMs)<n>We construct a new dataset of over 6,000 bilingual pairs across 16 languages using this methodology, demonstrating its effectiveness in revealing weaknesses even in state-of-the-art models.<n>Further experiments investigate the relationship between linguistic similarity and cross-lingual weaknesses, revealing that linguistically related languages share similar performance patterns.
  arXiv  Detail & Related papers  (2025-05-24T12:31:27Z)
• Evaluating Knowledge-based Cross-lingual Inconsistency in Large Language Models [16.942897938964638]
  Large Language Models (LLMs) have shown exceptional performance in various Natural Language Processing (NLP) tasks. Despite their successes, these models often exhibit significant inconsistencies when processing the same concepts across different languages. This study focuses on three primary questions: the existence of cross-lingual inconsistencies in LLMs, the specific aspects in which these inconsistencies manifest, and the correlation between cross-lingual consistency and multilingual capabilities of LLMs.
  arXiv  Detail & Related papers  (2024-07-01T15:11:37Z)
• Pre-Trained Language-Meaning Models for Multilingual Parsing and Generation [14.309869321407522]
  We introduce multilingual pre-trained language-meaning models based on Discourse Representation Structures (DRSs) Since DRSs are language neutral, cross-lingual transfer learning is adopted to further improve the performance of non-English tasks. automatic evaluation results show that our approach achieves the best performance on both the multilingual DRS parsing and DRS-to-text generation tasks.
  arXiv  Detail & Related papers  (2023-05-31T19:00:33Z)
• Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models [73.11488464916668]
  This study investigates the dynamics of the multilingual pretraining process. We probe checkpoints taken from throughout XLM-R pretraining, using a suite of linguistic tasks. Our analysis shows that the model achieves high in-language performance early on, with lower-level linguistic skills acquired before more complex ones.
  arXiv  Detail & Related papers  (2022-05-24T03:35:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.