Incentivizing Tool-augmented Thinking with Images for Medical Image Analysis

• URL: http://arxiv.org/abs/2512.14157v1
• Date: Tue, 16 Dec 2025 07:37:23 GMT
• Title: Incentivizing Tool-augmented Thinking with Images for Medical Image Analysis
• Authors: Yankai Jiang, Yujie Zhang, Peng Zhang, Yichen Li, Jintai Chen, Xiaoming Shi, Shihui Zhen,
• Abstract summary: We introduce Ophiuchus, a versatile, tool-augmented framework that equips an MLLM to decide when additional visual evidence is needed.<n>Our approach illuminates a path toward medical AI agents that can genuinely "think with images" through tool-integrated reasoning.
• Score: 35.90026194642237
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent reasoning based medical MLLMs have made progress in generating step by step textual reasoning chains. However, they still struggle with complex tasks that necessitate dynamic and iterative focusing on fine-grained visual regions to achieve precise grounding and diagnosis. We introduce Ophiuchus, a versatile, tool-augmented framework that equips an MLLM to (i) decide when additional visual evidence is needed, (ii) determine where to probe and ground within the medical image, and (iii) seamlessly weave the relevant sub-image content back into an interleaved, multimodal chain of thought. In contrast to prior approaches limited by the performance ceiling of specialized tools, Ophiuchus integrates the model's inherent grounding and perception capabilities with external tools, thereby fostering higher-level reasoning. The core of our method is a three-stage training strategy: cold-start training with tool-integrated reasoning data to achieve basic tool selection and adaptation for inspecting key regions; self-reflection fine-tuning to strengthen reflective reasoning and encourage revisiting tool outputs; and Agentic Tool Reinforcement Learning to directly optimize task-specific rewards and emulate expert-like diagnostic behavior. Extensive experiments show that Ophiuchus consistently outperforms both closed-source and open-source SOTA methods across diverse medical benchmarks, including VQA, detection, and reasoning-based segmentation. Our approach illuminates a path toward medical AI agents that can genuinely "think with images" through tool-integrated reasoning. Datasets, codes, and trained models will be released publicly.

Related papers

• MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learning [53.37068897861388]
  MedSAM-Agent is a framework that reformulates interactive segmentation as a multi-step autonomous decision-making process.<n>We develop a two-stage training pipeline that integrates multi-turn, end-to-end outcome verification.<n>Experiments across 6 medical modalities and 21 datasets demonstrate that MedSAM-Agent achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-02-03T09:47:49Z)
• Towards Agentic Intelligence for Materials Science [73.4576385477731]
  This survey advances a unique pipeline-centric view that spans from corpus curation and pretraining to goal-conditioned agents interfacing with simulation and experimental platforms.<n>To bridge communities and establish a shared frame of reference, we first present an integrated lens that aligns terminology, evaluation, and workflow stages across AI and materials science.
  arXiv  Detail & Related papers  (2026-01-29T23:48:43Z)
• MEDVISTAGYM: A Scalable Training Environment for Thinking with Medical Images via Tool-Integrated Reinforcement Learning [25.75780053067891]
  Vision language models (VLMs) achieve strong performance on general image understanding but struggle to think with medical images.<n>We introduce MedVistaGym, a scalable and interactive training environment that incentivizes tool-integrated visual reasoning for medical image analysis.
  arXiv  Detail & Related papers  (2026-01-12T00:11:10Z)
• Code-in-the-Loop Forensics: Agentic Tool Use for Image Forgery Detection [59.04089915447622]
  ForenAgent is an interactive IFD framework that enables MLLMs to autonomously generate, execute, and refine Python-based low-level tools around the detection objective.<n>Inspired by human reasoning, we design a dynamic reasoning loop comprising global perception, local focusing, iterative probing, and holistic adjudication.<n>Experiments show that ForenAgent exhibits emergent tool-use competence and reflective reasoning on challenging IFD tasks.
  arXiv  Detail & Related papers  (2025-12-18T08:38:44Z)
• MedEyes: Learning Dynamic Visual Focus for Medical Progressive Diagnosis [17.59077756990045]
  MedEyes is a reinforcement learning framework that dynamically models clinician-style diagnostic reasoning.<n>It emulates the diagnostic process through a dual-mode exploration strategy, scanning for systematic abnormality localization and drilling for detailed regional analysis.<n>Experiments demonstrate that MedEyes achieves an average performance improvement of +8.5% across multiple medical VQA benchmarks.
  arXiv  Detail & Related papers  (2025-11-27T01:47:43Z)
• MedAlign: A Synergistic Framework of Multimodal Preference Optimization and Federated Meta-Cognitive Reasoning [52.064286116035134]
  We develop MedAlign, a framework to ensure visually accurate LVLM responses for Medical Visual Question Answering (Med-VQA)<n>We first propose a multimodal Direct Preference Optimization (mDPO) objective to align preference learning with visual context.<n>We then design a Retrieval-Aware Mixture-of-Experts (RA-MoE) architecture that utilizes image and text similarity to route queries to a specialized and context-augmented LVLM.
  arXiv  Detail & Related papers  (2025-10-24T02:11:05Z)
• End-to-End Agentic RAG System Training for Traceable Diagnostic Reasoning [52.12425911708585]
  Deep-DxSearch is an agentic RAG system trained end-to-end with reinforcement learning (RL)<n>In Deep-DxSearch, we first construct a large-scale medical retrieval corpus comprising patient records and reliable medical knowledge sources.<n> Experiments demonstrate that our end-to-end RL training framework consistently outperforms prompt-engineering and training-free RAG approaches.
  arXiv  Detail & Related papers  (2025-08-21T17:42:47Z)
• EndoAgent: A Memory-Guided Reflective Agent for Intelligent Endoscopic Vision-to-Decision Reasoning [6.96058549084651]
  EndoAgent is a memory-guided agent for vision-to-decision endoscopic analysis.<n>It integrates iterative reasoning with adaptive tool selection and collaboration.<n>It consistently outperforms both general and medical multimodal models.
  arXiv  Detail & Related papers  (2025-08-10T11:02:57Z)
• Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applications [59.721265428780946]
  Large Language Models (LLMs) in medicine have enabled impressive capabilities, yet a critical gap remains in their ability to perform systematic, transparent, and verifiable reasoning.<n>This paper provides the first systematic review of this emerging field.<n>We propose a taxonomy of reasoning enhancement techniques, categorized into training-time strategies and test-time mechanisms.
  arXiv  Detail & Related papers  (2025-08-01T14:41:31Z)
• AURA: A Multi-Modal Medical Agent for Understanding, Reasoning & Annotation [0.8397730500554048]
  AURA is the first visual linguistic explainability agent designed specifically for comprehensive analysis, explanation, and evaluation of medical images.<n>AURA represents a significant advancement toward more transparent, adaptable, and clinically aligned AI systems.
  arXiv  Detail & Related papers  (2025-07-22T18:24:18Z)
• GEMeX-RMCoT: An Enhanced Med-VQA Dataset for Region-Aware Multimodal Chain-of-Thought Reasoning [60.03671205298294]
  Medical visual question answering aims to support clinical decision-making by enabling models to answer natural language questions based on medical images.<n>Current methods still suffer from limited answer reliability and poor interpretability.<n>This work first proposes a Region-Aware Multimodal Chain-of-Thought dataset, in which the process of producing an answer is preceded by a sequence of intermediate reasoning steps.
  arXiv  Detail & Related papers  (2025-06-22T08:09:58Z)
• Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search [41.81463064393831]
  Multimodal large language models (MLLMs) have begun to demonstrate robust reasoning capabilities on general tasks, yet their application in the medical domain remains in its early stages.<n>We propose Mentor-Intern Collaborative Search (MICS), a novel reasoning-path searching scheme to generate rigorous and effective medical CoT data.<n>We construct MMRP, a multi-task medical reasoning dataset with ranked difficulty, and Chiron-o1, a new medical MLLM devised via a curriculum learning strategy.
  arXiv  Detail & Related papers  (2025-06-20T12:51:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.