InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation

• URL: http://arxiv.org/abs/2608.22990v1
• Date: Mon, 24 Aug 2026 08:53:41 GMT
• Title: InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation
• Abstract summary: We introduce InstructMove, a text-indispensable benchmark for instruction-following manipulation.<n>InstructMove supports a train-eval protocol with InstructMove training data and held-out evaluation tasks.<n>Experiments with representative VLA policies show that InstructMove provides a controlled testbed for diagnosing visual shortcuts.
• Score: 24.530150557492934
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language-action (VLA) models have made general-purpose robot manipulation increasingly plausible by conditioning robot actions on natural-language instructions. A key test of such generality is whether policies actually follow language instructions. Yet many manipulation benchmarks leave this ability underdetermined: the intended object or destination is often visually salient or uniquely feasible, allowing policies to succeed without grounding the instruction. We argue that instruction-following evaluation should be text-indispensable: multiple actions should be visually and physically plausible, while only one should be consistent with the language instruction. We introduce InstructMove, a text-indispensable benchmark for instruction-following manipulation. InstructMove instantiates this principle in pick-and-place scenes with semantic distractors, decomposing instruction following into category identification, attribute discrimination, spatial reasoning, and compositional pick-and-place. InstructMove supports a train-eval protocol with InstructMove training data and held-out evaluation tasks, with additional diagnostics for language dependence. Experiments with representative VLA policies show that InstructMove provides a controlled testbed for diagnosing visual shortcuts and that InstructMove simulation data can improve real-world instruction-following manipulation performance. Code: https://github.com/HorizonRobotics/RoboOrchardSim

Related papers

• Natural Language Instructions for Scene-Responsive Human-in-the-Loop Motion Planning in Autonomous Driving using Vision-Language-Action Models [2.2727733134290813]
  doScenes is the first real-world dataset linking free-form instructions to nuScenes ground-truth motion.<n>We adapt OpenEMMA, an open-source MLLM-based end-to-end driving framework.<n>We integrate doScenes directives as passenger-style prompts within OpenEMMA's vision-language interface.
  arXiv  Detail & Related papers  (2026-02-04T03:44:56Z)
• Do What? Teaching Vision-Language-Action Models to Reject the Impossible [53.40183895299108]
  Vision-Language-Action (VLA) models have demonstrated strong performance on a range of robotic tasks.<n>We propose Instruct-Verify-and-Act (IVA), a framework that detects when an instruction cannot be executed due to a false premise.<n>Our experiments show that IVA improves false premise detection accuracy by 97.56% over baselines.
  arXiv  Detail & Related papers  (2025-08-22T10:54:33Z)
• Improving Instruction-Following in Language Models through Activation Steering [58.876600545898675]
  We derive instruction-specific vector representations from language models and use them to steer models accordingly.<n>We demonstrate how this method can enhance model adherence to constraints such as output format, length, and word inclusion.<n>Our findings demonstrate that activation steering offers a practical and scalable approach for fine-grained control in language generation.
  arXiv  Detail & Related papers  (2024-10-15T08:38:20Z)
• Spatially-Aware Speaker for Vision-and-Language Navigation Instruction Generation [8.931633531104021]
  SAS (Spatially-Aware Speaker) is an instruction generator that uses both structural and semantic knowledge of the environment to produce richer instructions. Our method outperforms existing instruction generation models, evaluated using standard metrics.
  arXiv  Detail & Related papers  (2024-09-09T13:12:11Z)
• NaturalVLM: Leveraging Fine-grained Natural Language for Affordance-Guided Visual Manipulation [21.02437461550044]
  Many real-world tasks demand intricate multi-step reasoning. We introduce a benchmark, NrVLM, comprising 15 distinct manipulation tasks. We propose a novel learning framework that completes the manipulation task step-by-step according to the fine-grained instructions.
  arXiv  Detail & Related papers  (2024-03-13T09:12:16Z)
• Answer is All You Need: Instruction-following Text Embedding via Answering the Question [41.727700155498546]
  This paper offers a new viewpoint, which treats the instruction as a question about the input text and encodes the expected answers to obtain the representation accordingly. Specifically, we propose InBedder that instantiates this embed-via-answering idea by only fine-tuning language models on abstractive question answering tasks.
  arXiv  Detail & Related papers  (2024-02-15T01:02:41Z)
• VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use [49.574651930395305]
  VisIT-Bench is a benchmark for evaluation of instruction-following vision-language models. Our dataset comprises 592 test queries, each with a human-authored instruction-conditioned caption. We quantify quality gaps between models and references using both human and automatic evaluations.
  arXiv  Detail & Related papers  (2023-08-12T15:27:51Z)
• Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Control [58.06223121654735]
  We show a method that taps into joint image- and goal- conditioned policies with language using only a small amount of language data. Our method achieves robust performance in the real world by learning an embedding from the labeled data that aligns language not to the goal image. We show instruction following across a variety of manipulation tasks in different scenes, with generalization to language instructions outside of the labeled data.
  arXiv  Detail & Related papers  (2023-06-30T20:09:39Z)
• Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model [63.66204449776262]
  Instruct2Act is a framework that maps multi-modal instructions to sequential actions for robotic manipulation tasks. Our approach is adjustable and flexible in accommodating various instruction modalities and input types. Our zero-shot method outperformed many state-of-the-art learning-based policies in several tasks.
  arXiv  Detail & Related papers  (2023-05-18T17:59:49Z)
• Modular Framework for Visuomotor Language Grounding [57.93906820466519]
  Natural language instruction following tasks serve as a valuable test-bed for grounded language and robotics research. We propose the structuring of language, acting, and visual tasks into separate modules that can be trained independently.
  arXiv  Detail & Related papers  (2021-09-05T20:11:53Z)
• Translating Natural Language Instructions to Computer Programs for Robot Manipulation [0.6629765271909505]
  We propose translating the natural language instruction to a Python function which queries the scene by accessing the output of the object detector. We show that the proposed method performs better than training a neural network to directly predict the robot actions.
  arXiv  Detail & Related papers  (2020-12-26T07:57:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.