VisTIRA: Closing the Image-Text Modality Gap in Visual Math Reasoning via Structured Tool Integration

• URL: http://arxiv.org/abs/2601.14440v1
• Date: Tue, 20 Jan 2026 19:54:49 GMT
• Title: VisTIRA: Closing the Image-Text Modality Gap in Visual Math Reasoning via Structured Tool Integration
• Authors: Saeed Khaki, Ashudeep Singh, Nima Safaei, Kamal Ginotra,
• Abstract summary: Vision-language models (VLMs) lag behind text-only language models on mathematical reasoning when the same problems are presented as images rather than text.<n>We introduce VisTIRA, a tool-integrated reasoning framework that enables structured problem solving by decomposing a given math problem (as an image) into natural language rationales.<n>We show that tool-integrated supervision improves image-based reasoning, and OCR grounding can further narrow the gap for smaller models.
• Score: 2.7403985180660784
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) lag behind text-only language models on mathematical reasoning when the same problems are presented as images rather than text. We empirically characterize this as a modality gap: the same question in text form yields markedly higher accuracy than its visually typeset counterpart, due to compounded failures in reading dense formulas, layout, and mixed symbolic-diagrammatic context. First, we introduce VisTIRA (Vision and Tool-Integrated Reasoning Agent), a tool-integrated reasoning framework that enables structured problem solving by iteratively decomposing a given math problem (as an image) into natural language rationales and executable Python steps to determine the final answer. Second, we build a framework to measure and improve visual math reasoning: a LaTeX-based pipeline that converts chain-of-thought math corpora (e.g., NuminaMath) into challenging image counterparts, and a large set of synthetic tool-use trajectories derived from a real-world, homework-style image dataset (called SnapAsk) for fine-tuning VLMs. Our experiments show that tool-integrated supervision improves image-based reasoning, and OCR grounding can further narrow the gap for smaller models, although its benefit diminishes at scale. These findings highlight that modality gap severity inversely correlates with model size, and that structured reasoning and OCR-based grounding are complementary strategies for advancing visual mathematical reasoning.

Related papers

• Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models [84.78794648147608]
  A persistent geometric anomaly, the Modality Gap, remains.<n>Prior approaches to bridge this gap are largely limited by oversimplified isotropic assumptions.<n>We propose the Fixed-frame Modality Gap Theory, which decomposes the modality gap into stable biases and anisotropic residuals.<n>We then introduce ReAlign, a training-free modality alignment strategy.
  arXiv  Detail & Related papers  (2026-02-02T13:59:39Z)
• When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought [118.71264263478083]
  We propose MIRA, a new benchmark designed to evaluate models in scenarios where generating intermediate visual images is essential for successful reasoning.<n>We include 546 multimodal problems, annotated with intermediate visual images and final answers.
  arXiv  Detail & Related papers  (2025-11-04T18:00:51Z)
• MathCanvas: Intrinsic Visual Chain-of-Thought for Multimodal Mathematical Reasoning [58.776297011268845]
  We present a comprehensive framework designed to endow unified Large Multimodal Models with intrinsic VCoT capabilities for mathematics.<n>Our model, BAGEL-canvas, trained under this framework, achieves an 86% relative improvement over strong LMM baselines.<n>Our work provides a complete toolkit-framework, datasets, and benchmark-to unlock complex, human-like visual-aided reasoning in LMMs.
  arXiv  Detail & Related papers  (2025-10-16T17:58:58Z)
• CodePlot-CoT: Mathematical Visual Reasoning by Thinking with Code-Driven Images [69.93976232543066]
  We propose CodePlot-CoT, a code-driven Chain-of-Thought paradigm for "thinking with images" in mathematics.<n>To achieve this, we first construct Math-VR, the first large-scale, bilingual dataset and benchmark for Mathematics problems with Visual Reasoning.<n>Our model achieves up to 21% increase over base model on our new benchmark, fully validating the efficacy of our proposed code-driven reasoning paradigm.
  arXiv  Detail & Related papers  (2025-10-13T17:59:55Z)
• VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMs [31.007061220012954]
  We present groundingMath, a curated benchmark of 1,800 high-quality K-12 mathematics problems in which all candidate answers are diagrams with subtle visual similarities.<n>A comprehensive evaluation of state-of-the-art LMMs, covering both leading closed-source systems and widely adopted open-source models, reveals a consistent decline in accuracy as inter-image similarity increases.<n>We explore three alignment-oriented strategies, spanning training-free approaches and finetuning, to achieve substantial accuracy gains.
  arXiv  Detail & Related papers  (2025-06-07T09:24:13Z)
• Visualizing Thought: Conceptual Diagrams Enable Robust Planning in LMMs [59.66595230543127]
  Conceptual diagrams externalize mental models, abstracting irrelevant details to efficiently capture how entities interact.<n>Large Language Models (LLMs) and Large MultiModal Models (LMMs) predominantly reason through text.<n>We propose Visual Thinking, a generalizable framework that enables LMMs to reason through multiple chains of self-generated conceptual diagrams.
  arXiv  Detail & Related papers  (2025-03-14T18:27:02Z)
• The Role of Visual Modality in Multimodal Mathematical Reasoning: Challenges and Insights [26.85150689408895]
  We show that existing multimodal mathematical models minimally leverage visual information.<n>We attribute this to the dominance of textual information and answer options that inadvertently guide the model to correct answers.<n>In testing leading models, their failure to detect subtle visual differences suggests limitations in current visual perception capabilities.
  arXiv  Detail & Related papers  (2025-03-06T07:29:33Z)
• Open Eyes, Then Reason: Fine-grained Visual Mathematical Understanding in MLLMs [62.875934732547435]
  Current large language models (MLLMs) often underperform on mathematical problem-solving tasks that require fine-grained visual understanding.<n>In this paper, we evaluate the visual grounding capabilities of state-of-the-art MLLMs and reveal a significant negative correlation between visual grounding accuracy and problem-solving performance.<n>We propose a novel approach, SVE-Math, featuring a geometric-grounded vision encoder and a feature router that dynamically adjusts the contribution of hierarchical visual feature maps.
  arXiv  Detail & Related papers  (2025-01-11T04:08:44Z)
• Chain of Images for Intuitively Reasoning [23.692458865558486]
  We present a Chain of Images (CoI) approach to convert complex language reasoning problems to simple pattern recognition. We have developed a CoI evaluation dataset encompassing 15 distinct domains where images can intuitively aid problem-solving. In supporting our CoI reasoning, we introduce a symbolic multimodal large language model (SyMLLM) that generates images strictly based on language instructions.
  arXiv  Detail & Related papers  (2023-11-09T11:14:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.