Robust Skills, Brittle Grounding: Diagnosing Restricted Generalization in Vision-Language Action Policies via Multi-Object Picking

• URL: http://arxiv.org/abs/2602.24143v1
• Date: Fri, 27 Feb 2026 16:20:04 GMT
• Title: Robust Skills, Brittle Grounding: Diagnosing Restricted Generalization in Vision-Language Action Policies via Multi-Object Picking
• Abstract summary: Vision-language action (VLA) policies often report strong manipulation benchmark performance with relatively few demonstrations.<n>We present a controlled multi-object picking study that progressively increases object placement variability up to full workspace randomization.<n>We find that for representative VLA policies, including SmolVLA and $_0.5$, execution of the manipulation primitive remains substantially more reliable than instruction-conditioned task success.
• Score: 6.713814849613683
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language action (VLA) policies often report strong manipulation benchmark performance with relatively few demonstrations, but it remains unclear whether this reflects robust language-to-object grounding or reliance on object--location correlations that do not transfer beyond the training distribution. We present a controlled multi-object picking study that progressively increases object placement variability up to full workspace randomization and evaluates held-out object--location pairings that break familiar associations without increasing spatial difficulty. Across these stress tests and data scaling, we find that for representative VLA policies, including SmolVLA and $π_{0.5}$, execution of the manipulation primitive remains substantially more reliable than instruction-conditioned task success in harder regimes, suggesting that manipulation skill acquisition is decoupled from instruction following. We recommend augmenting manipulation benchmarks with task ladders and decomposed metrics that separately measure primitive execution and instruction-conditioned success to better diagnose instruction-grounded generalization.

Related papers

• ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuning [53.17987412112712]
  Reinforcement Learning (RL) has demonstrated significant potential for improving Vision-Language-Action (VLA) models on complex manipulation tasks.<n>In this work, we first investigate the exploration stagnation bottleneck in current VLA-RL frameworks.<n>We introduce RL Exploration Token (ExToken), a simple yet general framework that condition VLA policies on discrete behavioral priors.
  arXiv  Detail & Related papers  (2026-07-14T16:04:41Z)
• RoVLA: Multi-Consistency Constraints for Robust Vision-Language-Action Models [41.35967018882011]
  Vision-Language-Action (VLA) models have shown strong performance on embodied manipulation.<n>They remain brittle under visual observation changes, paraphrased language instructions, and compounded perturbations.<n>We propose RoVLA, a robust vision-language-action framework with multi-consistency constraints.
  arXiv  Detail & Related papers  (2026-05-19T11:10:20Z)
• TAG: Target-Agnostic Guidance for Stable Object-Centric Inference in Vision-Language-Action Models [59.13964209628383]
  Vision-Language--Action (VLA) policies have shown strong progress in mapping language instructions and visual observations to robotic actions, yet their reliability degrades in cluttered scenes with distractors.<n>We propose TAG (Target-Agnostic Guidance), a simple inference-time guidance mechanism that explicitly reduces distractor- and appearance-induced bias in VLA policies.<n>We evaluate TAG on standard manipulation benchmarks, including LIBERO, LIBERO-Plus, and VLABench, where it consistently improves robustness under clutter and reduces near-miss and wrong-object executions.
  arXiv  Detail & Related papers  (2026-03-25T17:56:32Z)
• Language-Grounded Decoupled Action Representation for Robotic Manipulation [78.42228162226839]
  We propose the Language-Grounded Decoupled Action Representation (LaDA) framework to connect perception and control.<n>LaDA introduces a fine-grained intermediate layer of three interpretable action primitives--translation, rotation, and gripper control--providing explicit semantic structure for low-level actions.<n>It further employs a semantic-guided soft-label contrastive learning objective to align similar action primitives across tasks, enhancing generalization and motion consistency.
  arXiv  Detail & Related papers  (2026-03-13T13:08:26Z)
• Act, Think or Abstain: Complexity-Aware Adaptive Inference for Vision-Language-Action Models [7.802379200026965]
  We propose an adaptive framework that dynamically routes VLA execution based on the complexity of the perceived state.<n>Our approach transforms the VLA's vision-language backbone into an active detection tool by projecting latent embeddings into an ensemble of parametric and non-parametric estimators.
  arXiv  Detail & Related papers  (2026-03-05T13:14:41Z)
• When to Act, Ask, or Learn: Uncertainty-Aware Policy Steering [10.01278648231868]
  Policy steering is an emerging way to adapt robot behaviors at deployment-time.<n> Vision-Language Models (VLMs) are promising general-purpose verifiers due to their reasoning capabilities.<n>We propose uncertainty-aware policy steering (UPS), a framework that jointly reasons about semantic task uncertainty and low-level action feasibility.
  arXiv  Detail & Related papers  (2026-02-25T23:23:22Z)
• Generalizing from References using a Multi-Task Reference and Goal-Driven RL Framework [12.131501436717969]
  We introduce a unified multi-task Reinforcement Learning framework for learning humanoid behaviors from human motion.<n>A single goal-conditioned policy is trained jointly on two tasks that share the same observation and action spaces.<n>By co-optimizing these objectives within a shared formulation, the policy acquires structured, human-like motor skills from dense reference supervision.
  arXiv  Detail & Related papers  (2026-02-23T21:25:06Z)
• Scaling Verification Can Be More Effective than Scaling Policy Learning for Vision-Language-Action Alignment [58.93227458806748]
  CoVer-VLA is a hierarchical test-time verification pipeline using the trained verifier.<n>Our framework precomputes a diverse set of rephrased instructions from a Vision-Language-Model.<n>It repeatedly generates action candidates for each instruction, and then uses the verifier to select the optimal high-level prompt and low-level action chunks.
  arXiv  Detail & Related papers  (2026-02-12T18:59:59Z)
• Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization [65.37179698521766]
  Vision-Language-Action (VLA) models show strong promise for generalist robot control.<n>It remains unclear whether -- and under what conditions -- the standard "scale data" recipe translates to robotics.<n>We present a systematic, controlled study of VLA scaling that revisits core training choices for pretraining across diverse robots.
  arXiv  Detail & Related papers  (2026-02-10T12:25:43Z)
• LIBERO-X: Robustness Litmus for Vision-Language-Action Models [32.29541801424534]
  This work systematically rethinks VLA benchmarking from both evaluation and data perspectives.<n>We introduce LIBERO-X, a benchmark featuring a hierarchical evaluation protocol with progressive difficulty levels targeting three core capabilities.<n> Experiments with representative VLA models reveal significant performance drops under cumulative perturbations.
  arXiv  Detail & Related papers  (2026-02-06T09:59:12Z)
• Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach [78.4812458793128]
  We propose textbfTACO, a test-time-scaling framework that applies a lightweight pseudo-count estimator as a high-fidelity verifier of action chunks.<n>Our method resembles the classical anti-exploration principle in offline reinforcement learning (RL), and being gradient-free, it incurs significant computational benefits.
  arXiv  Detail & Related papers  (2025-12-02T14:42:54Z)
• FOSSIL: Harnessing Feedback on Suboptimal Samples for Data-Efficient Generalisation with Imitation Learning for Embodied Vision-and-Language Tasks [45.65159253753118]
  This work explores how agents trained with imitation learning can learn robust representations from both optimal and suboptimal demonstrations.<n>We provide language feedback embeddings as part of the input sequence into a Transformer-based policy.<n>We test our approach on a range of embodied Vision-and-Language tasks in our custom BabyAI-XGen environment.
  arXiv  Detail & Related papers  (2025-10-13T11:55:21Z)
• Do What? Teaching Vision-Language-Action Models to Reject the Impossible [53.40183895299108]
  Vision-Language-Action (VLA) models have demonstrated strong performance on a range of robotic tasks.<n>We propose Instruct-Verify-and-Act (IVA), a framework that detects when an instruction cannot be executed due to a false premise.<n>Our experiments show that IVA improves false premise detection accuracy by 97.56% over baselines.
  arXiv  Detail & Related papers  (2025-08-22T10:54:33Z)
• COMBO-Grasp: Learning Constraint-Based Manipulation for Bimanual Occluded Grasping [56.907940167333656]
  Occluded robot grasping is where the desired grasp poses are kinematically infeasible due to environmental constraints such as surface collisions.<n>Traditional robot manipulation approaches struggle with the complexity of non-prehensile or bimanual strategies commonly used by humans.<n>We introduce Constraint-based Manipulation for Bimanual Occluded Grasping (COMBO-Grasp), a learning-based approach which leverages two coordinated policies.
  arXiv  Detail & Related papers  (2025-02-12T01:31:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.