One Word, Different Action: A Real-Robot Benchmark for Language-Conditioned Embodied Reasoning

• URL: http://arxiv.org/abs/2609.05260v1
• Date: Fri, 04 Sep 2026 15:21:58 GMT
• Title: One Word, Different Action: A Real-Robot Benchmark for Language-Conditioned Embodied Reasoning
• Abstract summary: We introduce One Word, Different Action, a real-robot benchmark built on physical decision states and executable actions.<n>Experiments show that modern models are near saturation on single-constraint instruction changes, yet several models degrade noticeably when multiple task constraints must be integrated into one executable decision.
• Score: 5.503904431371255
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Natural-language instruction changes can directly alter robot behavior. A reliable embodied system should preserve its action when the task is unchanged and update it correctly when the task itself changes. We introduce One Word, Different Action, a real-robot benchmark built on physical decision states and executable actions, using task-preserving and task-changing instruction pairs to jointly evaluate Decision Invariance and Decision Sensitivity, with further evaluation under multi-constraint reasoning and real-RGB grounding. Experiments show that modern models are near saturation on single-constraint instruction changes, yet several models degrade noticeably when multiple task constraints must be integrated into one executable decision. These results suggest that the more salient remaining challenge is no longer recognizing an isolated instruction change, but reliably composing multiple task requirements into a correct robot action decision.

Related papers

• IMBench: A Benchmark for Intuitive Robotic Manipulation [11.843368723732679]
  Humans combine reasoning and motor control to solve complex manipulation tasks under diverse constraints.<n>Existing benchmarks fail to capture this integration.<n>We introduce IMBENCH, a benchmark designed to evaluate intuitive manipulation as an integrated capability spanning perception, physical reasoning, action generation, and iterative execution.
  arXiv  Detail & Related papers  (2026-07-17T05:34:29Z)
• RoboWits: Unexpected Challenges for Robotic Creative Problem Solving [58.58727722988084]
  We introduce RoboWits, a bi-manual robotic benchmark designed to evaluate cognitive reasoning, creative tool use, and robustness to unexpected conditions.<n>Using the pipeline, we curated 30 diverse seed tasks and 208 tasks with mutations and graded difficulty across geometry, material, and assembly-based reasoning.<n>Our results reveal a significant performance gap: while pre-trained VLAs exhibit preliminary success on seed tasks after single-task fine-tuning, they struggle to perform on mutated tasks.
  arXiv  Detail & Related papers  (2026-05-28T17:57:15Z)
• RoboPlayground: Democratizing Robotic Evaluation through Structured Physical Domains [19.460905215536254]
  Evaluation of robotic manipulation systems has largely relied on fixed benchmarks authored by experts.<n>We argue that evaluating modern manipulation policies requires reframing evaluation as a language-driven process.<n>We present RoboPlayground, a framework that enables users to author executable manipulation tasks using natural language.
  arXiv  Detail & Related papers  (2026-04-06T22:42:05Z)
• CORAL: Scalable Multi-Task Robot Learning via LoRA Experts [49.759823970016974]
  Vision-Language-Action (VLA) models in real-world robotics expose a core multi-task learning challenge.<n>Maintaining a separate full checkpoint per task is often storage- and deployment-prohibitive.<n>We present CORAL, a backbone- and-agnostic framework designed primarily to mitigate multi-task interference.
  arXiv  Detail & Related papers  (2026-03-10T07:28:41Z)
• Enhancing Goal Inference via Correction Timing [2.444217495283211]
  Corrections offer a natural modality for people to provide feedback to a robot.<n>Most prior work on learning from corrections involves interpreting a correction as a new demonstration.<n>We investigate whether the timing of this decision can offer a useful signal for inferring task-relevant influences.
  arXiv  Detail & Related papers  (2026-02-20T20:38:47Z)
• Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations [76.79742393097358]
  Vision-Language Action (VLAs) models promise to extend the remarkable success of vision-language models (VLMs) to robotics.<n>Existing fine-tuning methods lack specificity, adapting the same set of parameters regardless of a task's visual, linguistic, and physical characteristics.<n>Inspired by functional specificity in neuroscience, we hypothesize that it is more effective to finetune sparse model representations specific to a given task.
  arXiv  Detail & Related papers  (2025-11-27T18:50:21Z)
• Embodied Red Teaming for Auditing Robotic Foundation Models [39.23336328126245]
  Embodied Red Teaming (ERT) generates diverse and challenging instructions to test language-conditioned robot models.<n>ERT uses automated red teaming techniques with Vision Language Models (VLMs) to create contextually grounded, difficult instructions.<n> Experimental results show that state-of-the-art language-conditioned robot models fail or behave unsafely on ERT-generated instructions.
  arXiv  Detail & Related papers  (2024-11-27T18:57:26Z)
• Closed Loop Interactive Embodied Reasoning for Robot Manipulation [17.732550906162192]
  Embodied reasoning systems integrate robotic hardware and cognitive processes to perform complex tasks.<n>We introduce a new modular Closed Loop Interactive Embodied Reasoning (CLIER) approach.<n>CLIER takes into account the measurements of non-visual object properties, changes in the scene caused by external disturbances as well as uncertain outcomes of robotic actions.
  arXiv  Detail & Related papers  (2024-04-23T16:33:28Z)
• Towards Robust Continual Learning with Bayesian Adaptive Moment Regularization [51.34904967046097]
  Continual learning seeks to overcome the challenge of catastrophic forgetting, where a model forgets previously learnt information. We introduce a novel prior-based method that better constrains parameter growth, reducing catastrophic forgetting. Results show that BAdam achieves state-of-the-art performance for prior-based methods on challenging single-headed class-incremental experiments.
  arXiv  Detail & Related papers  (2023-09-15T17:10:51Z)
• Instruction-driven history-aware policies for robotic manipulations [82.25511767738224]
  We propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations. We evaluate our method on the challenging RLBench benchmark and on a real-world robot.
  arXiv  Detail & Related papers  (2022-09-11T16:28:25Z)
• Autonomous Open-Ended Learning of Tasks with Non-Stationary Interdependencies [64.0476282000118]
  Intrinsic motivations have proven to generate a task-agnostic signal to properly allocate the training time amongst goals. While the majority of works in the field of intrinsically motivated open-ended learning focus on scenarios where goals are independent from each other, only few of them studied the autonomous acquisition of interdependent tasks. In particular, we first deepen the analysis of a previous system, showing the importance of incorporating information about the relationships between tasks at a higher level of the architecture. Then we introduce H-GRAIL, a new system that extends the previous one by adding a new learning layer to store the autonomously acquired sequences
  arXiv  Detail & Related papers  (2022-05-16T10:43:01Z)
• Towards More Generalizable One-shot Visual Imitation Learning [81.09074706236858]
  A general-purpose robot should be able to master a wide range of tasks and quickly learn a novel one by leveraging past experiences. One-shot imitation learning (OSIL) approaches this goal by training an agent with (pairs of) expert demonstrations. We push for a higher level of generalization ability by investigating a more ambitious multi-task setup.
  arXiv  Detail & Related papers  (2021-10-26T05:49:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.