LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models

• URL: http://arxiv.org/abs/2603.28301v1
• Date: Mon, 30 Mar 2026 11:27:34 GMT
• Title: LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models
• Abstract summary: We introduce LIBERO-Para, a benchmark for fine-grained analysis of linguistic generalization.<n>We observe consistent performance degradation of 22-52 pp under paraphrasing.<n>We propose PRIDE, a metric that quantifies paraphrase difficulty using semantic and syntactic factors.
• Score: 15.955490895662384
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Vision-Language-Action (VLA) models achieve strong performance in robotic manipulation by leveraging pre-trained vision-language backbones. However, in downstream robotic settings, they are typically fine-tuned with limited data, leading to overfitting to specific instruction formulations and leaving robustness to paraphrased instructions underexplored. To study this gap, we introduce LIBERO-Para, a controlled benchmark that independently varies action expressions and object references for fine-grained analysis of linguistic generalization. Across seven VLA configurations (0.6B-7.5B), we observe consistent performance degradation of 22-52 pp under paraphrasing. This degradation is primarily driven by object-level lexical variation: even simple synonym substitutions cause large drops, indicating reliance on surface-level matching rather than semantic grounding. Moreover, 80-96% of failures arise from planning-level trajectory divergence rather than execution errors, showing that paraphrasing disrupts task identification. Binary success rate treats all paraphrases equally, obscuring whether models perform consistently across difficulty levels or rely on easier cases. To address this, we propose PRIDE, a metric that quantifies paraphrase difficulty using semantic and syntactic factors. Our benchmark and corresponding code are available at: https://github.com/cau-hai-lab/LIBERO-Para

Related papers

• Grounded Semantic Re-Binding for Robust Instruction Generalization in Vision-Language-Action Models [20.8047896631941]
  Vision-Language-Action (VLA) models excel in robotic manipulation but suffer catastrophic performance drops when canonical instructions are paraphrased.<n>We show that current VLAs successfully retain the correct task identity internally.<n>We propose Grounded Semantic Re-binding (GSR), an elegant intervention that bypasses unstable joint routing.
  arXiv  Detail & Related papers  (2026-08-03T17:02:04Z)
• Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluation [53.844308305341166]
  We propose textbfAgentic ASR, a closed-loop framework that combines a single-pass ASR front-end with semantic correction, intent routing, and reasoning-based editing.<n> Experiments on multilingual, named-entity-intensive, and code-switching benchmarks show that iterative interaction consistently reduces semantic errors.
  arXiv  Detail & Related papers  (2026-05-28T06:23:31Z)
• TAG: Target-Agnostic Guidance for Stable Object-Centric Inference in Vision-Language-Action Models [59.13964209628383]
  Vision-Language--Action (VLA) policies have shown strong progress in mapping language instructions and visual observations to robotic actions, yet their reliability degrades in cluttered scenes with distractors.<n>We propose TAG (Target-Agnostic Guidance), a simple inference-time guidance mechanism that explicitly reduces distractor- and appearance-induced bias in VLA policies.<n>We evaluate TAG on standard manipulation benchmarks, including LIBERO, LIBERO-Plus, and VLABench, where it consistently improves robustness under clutter and reduces near-miss and wrong-object executions.
  arXiv  Detail & Related papers  (2026-03-25T17:56:32Z)
• Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization [65.37179698521766]
  Vision-Language-Action (VLA) models show strong promise for generalist robot control.<n>It remains unclear whether -- and under what conditions -- the standard "scale data" recipe translates to robotics.<n>We present a systematic, controlled study of VLA scaling that revisits core training choices for pretraining across diverse robots.
  arXiv  Detail & Related papers  (2026-02-10T12:25:43Z)
• CORE: Context-Robust Remasking for Diffusion Language Models [51.59514489363897]
  We propose Context-Robust Remasking (CORE), a training-free framework for inference-time revision.<n>Rather than trusting static token probabilities, CORE identifies context-brittle tokens by probing their sensitivity to targeted masked-context perturbations.<n>On LLaDA-8B-Base, CORE delivers consistent improvements across reasoning and code benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.
  arXiv  Detail & Related papers  (2026-02-04T00:12:30Z)
• RoParQ: Paraphrase-Aware Alignment of Large Language Models Towards Robustness to Paraphrased Questions [0.0]
  Large Language Models (LLMs) often exhibit inconsistent behavior when answering paraphrased questions.<n>We introduce RoParQ, a benchmark to evaluate cross-paraphrase consistency in closed-book multiple-choice QA.<n>We also propose XParaCon, a novel evaluation metric that quantifies a model's robustness.
  arXiv  Detail & Related papers  (2025-11-26T16:40:53Z)
• LLMs Show Surface-Form Brittleness Under Paraphrase Stress Tests [0.0]
  Benchmark scores for Large Language Models (LLMs) can be inflated by memorization of test items or near duplicates.<n>We present a protocol that probes generalization by re-evaluating models on paraphrased versions of benchmark questions.
  arXiv  Detail & Related papers  (2025-10-08T06:04:33Z)
• Do What? Teaching Vision-Language-Action Models to Reject the Impossible [53.40183895299108]
  Vision-Language-Action (VLA) models have demonstrated strong performance on a range of robotic tasks.<n>We propose Instruct-Verify-and-Act (IVA), a framework that detects when an instruction cannot be executed due to a false premise.<n>Our experiments show that IVA improves false premise detection accuracy by 97.56% over baselines.
  arXiv  Detail & Related papers  (2025-08-22T10:54:33Z)
• Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Models [60.00178316095646]
  Sentence embedding is essential for many NLP tasks, with contrastive learning methods achieving strong performance using datasets like NLI.<n>Recent studies leverage large language models (LLMs) to generate sentence pairs, reducing annotation dependency.<n>We propose a method for controlling the generation direction of LLMs in the latent space. Unlike unconstrained generation, the controlled approach ensures meaningful semantic divergence.<n> Experiments on multiple benchmarks demonstrate that our method achieves new SOTA performance with a modest cost in ranking sentence synthesis.
  arXiv  Detail & Related papers  (2025-02-19T12:07:53Z)
• DECIDER: Leveraging Foundation Model Priors for Improved Model Failure Detection and Explanation [18.77296551727931]
  We propose DECIDER, a novel approach that leverages priors from large language models (LLMs) and vision-language models (VLMs) to detect failures in image models. DECIDER consistently achieves state-of-the-art failure detection performance, significantly outperforming baselines in terms of the overall Matthews correlation coefficient.
  arXiv  Detail & Related papers  (2024-08-01T07:08:11Z)
• Scalable Performance Analysis for Vision-Language Models [26.45624201546282]
  Joint vision-language models have shown great performance over a diverse set of tasks. Our paper introduces a more scalable solution that relies on already annotated benchmarks. We confirm previous findings that CLIP behaves like a bag of words model and performs better with nouns and verbs.
  arXiv  Detail & Related papers  (2023-05-30T06:40:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.