VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use

• URL: http://arxiv.org/abs/2505.19255v3
• Date: Wed, 11 Jun 2025 21:47:49 GMT
• Title: VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use
• Authors: Mingyuan Wu, Jingcheng Yang, Jize Jiang, Meitang Li, Kaizhuo Yan, Hanchao Yu, Minjia Zhang, Chengxiang Zhai, Klara Nahrstedt,
• Abstract summary: We introduce VTool-R1, the first framework that trains vision-language models to generate multimodal chains of thought.<n>VTool-R1 integrates Python-based visual editing tools into theReinforcement Learning Finetuning process.
• Score: 33.83255323522487
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement Learning Finetuning (RFT) has significantly advanced the reasoning capabilities of large language models (LLMs) by enabling long chains of thought, self-correction, and effective tool use. While recent works attempt to extend RFT to vision-language models (VLMs), these efforts largely produce text-only reasoning conditioned on static image inputs, falling short of true multimodal reasoning in the response. In contrast, test-time methods like Visual Sketchpad incorporate visual steps but lack training mechanisms. We introduce VTool-R1, the first framework that trains VLMs to generate multimodal chains of thought by interleaving text and intermediate visual reasoning steps. VTool-R1 integrates Python-based visual editing tools into the RFT process, enabling VLMs to learn when and how to generate visual reasoning steps that benefit final reasoning. Trained with outcome-based rewards tied to task accuracy, our approach elicits strategic visual tool use for reasoning without relying on process-based supervision. Experiments on structured visual question answering over charts and tables show that VTool-R1 enhances reasoning performance by teaching VLMs to "think with images" and generate multimodal chain of thoughts with tools.

Related papers

• Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback [33.127607245587576]
  We introduce a framework that enables MLLMs to learn complex visual reasoning from only raw images.<n>We demonstrate that this relative ease provides an ideal reward signal for optimization via Reinforcement Learning.<n>The RRVF-trained model not only outperforms existing MLLMs and supervised fine-tuning baselines but also exhibits superior generalization.
  arXiv  Detail & Related papers  (2025-07-28T12:21:19Z)
• Decoupled Visual Interpretation and Linguistic Reasoning for Math Problem Solving [57.22004912994658]
  Current large vision-language models (LVLMs) typically employ a connector module to link visual features with text embeddings of large language models (LLMs)<n>This paper proposes a paradigm shift: instead of training end-to-end vision-language reasoning models, we advocate for developing a decoupled reasoning framework.
  arXiv  Detail & Related papers  (2025-05-23T08:18:00Z)
• OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning [57.89304342666846]
  We introduce OpenThinkIMG, the first open-source, comprehensive end-to-end framework for tool-augmented LVLMs.<n>We propose a novel reinforcement learning framework V-ToolRL to train LVLMs to learn adaptive policies for invoking external vision tools.<n>V-ToolRL enables LVLMs to autonomously discover optimal tool-usage strategies.
  arXiv  Detail & Related papers  (2025-05-13T14:35:51Z)
• VIPER: Visual Perception and Explainable Reasoning for Sequential Decision-Making [21.61801132083334]
  VIPER is a novel framework for multimodal instruction-based planning.<n>It integrates VLM-based perception with LLM-based reasoning.<n>We show that VIPER significantly outperforms state-of-the-art visual instruction-based planners.
  arXiv  Detail & Related papers  (2025-03-19T11:05:42Z)
• Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models [127.38740043393527]
  We propose ViFT, a visual instruction-free fine-tuning framework for LVLMs.<n>We only require the text-only instructions and image caption data during training, to separately learn the task-solving and visual perception abilities.<n> Experimental results demonstrate that ViFT can achieve state-of-the-art performance on several visual reasoning and visual instruction following benchmarks.
  arXiv  Detail & Related papers  (2025-02-17T04:38:12Z)
• Instruction Tuning-free Visual Token Complement for Multimodal LLMs [51.138806401996696]
  multimodal large language models (MLLMs) have promised an elegant bridge between vision and language. We propose a Visual Token Complement framework (VTC) that helps MLLMs regain the missing visual features. Our VTC integrates text-to-image generation as a guide to identifying the text-irrelevant features, and a visual selector is then developed to generate complementary visual tokens.
  arXiv  Detail & Related papers  (2024-08-09T12:13:01Z)
• Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Training [24.989732666940153]
  Open-source multimodal large language models (MLLMs) excel in various tasks involving textual and visual inputs. MLLMs still struggle with complex multimodal mathematical reasoning, lagging behind proprietary models like GPT-4V(ision) and Gemini-Pro. We propose a two-step training pipeline VCAR, which emphasizes the Visual Reasoning training in addition to mathematical learning.
  arXiv  Detail & Related papers  (2024-04-22T21:59:35Z)
• CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Update [69.59482029810198]
  CLOVA is a Closed-Loop Visual Assistant that operates within a framework encompassing inference, reflection, and learning phases. Results demonstrate that CLOVA surpasses existing tool-usage methods by 5% in visual question answering and multiple-image reasoning, by 10% in knowledge tagging, and by 20% in image editing.
  arXiv  Detail & Related papers  (2023-12-18T03:34:07Z)
• VILA: On Pre-training for Visual Language Models [74.08039416548209]
  We study the design options for VLM pre-training through step-by-step controllable comparisons. We build VILA, a Visual Language model family that consistently outperforms the state-of-the-art models.
  arXiv  Detail & Related papers  (2023-12-12T18:58:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.