CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions

• URL: http://arxiv.org/abs/2608.28958v1
• Date: Sat, 29 Aug 2026 00:03:13 GMT
• Title: CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions
• Abstract summary: Chain-of-thought (CoT) reasoning has dramatically improved large language models (LLMs)<n>CoVA-SFT is a highly structured corpus of 51.9K samples containing over 222K multimodal reasoning steps across 5 distinct layout families and 17 complex tasks.<n>By providing explicit rationale formulations, agentic renderings, and verification loops, CoVA-SFT teaches multimodal language models to interleave text and visual abstractions.
• Score: 73.85100620647452
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Chain-of-thought (CoT) reasoning has dramatically improved large language models (LLMs) by allowing them to decompose problems into intermediate steps. While CoT is widely effective for linguistic tasks, text-only CoT forces models to serialize visual problems into awkward prose. Although architectural solutions exist to process visual inputs, the community lacks a massive, multi-step, self-corrected dataset to teach models how to build and maintain internal visual workspaces when solving purely textual reasoning problems. To address this limitation, we introduce CoVA-SFT, a highly structured corpus of 51.9K samples containing over 222K multimodal reasoning steps across 5 distinct layout families and 17 complex tasks, and CoVA-Bench, a companion benchmark of 1,700 held-out test samples spanning the same tasks for reproducible evaluation. By providing explicit rationale formulations, agentic renderings, and verification loops, CoVA-SFT teaches multimodal language models to interleave text and visual abstractions. We validate the dataset by demonstrating that models fine-tuned on CoVA-SFT outperform all interleaved CoT baselines by more than 2x on average on CoVA-Bench, though they still fall short of strong text-only CoT baselines, highlighting open challenges for future work.

Related papers

• Canvas-of-Thought: Grounding Reasoning via Mutable Structured States [18.74176136703898]
  Chain-of-Thought (CoT) prompting has significantly advanced the reasoning capabilities of Multimodal Large Language Models (MLLMs)<n> relying solely on linear text sequences remains a bottleneck for complex tasks.<n>We introduce textbf Canvas-of-Thought ( Canvas-CoT), a new paradigm for context-efficient multimodal reasoning.
  arXiv  Detail & Related papers  (2026-02-11T03:54:58Z)
• Table2LaTeX-RL: High-Fidelity LaTeX Code Generation from Table Images via Reinforced Multimodal Language Models [53.03670032402846]
  We address the task of table image to code generation, with the goal of automating the reconstruction of high-quality, publication-ready tables from visual inputs.<n>A central challenge of this task lies in accurately handling complex tables -- those with large sizes, deeply nested structures, and semantically rich or irregular cell content.<n>We propose a reinforced multimodal large language model (MLLM) framework, where a pre-trained MLLM is fine-tuned on a large-scale table-to-La dataset.
  arXiv  Detail & Related papers  (2025-09-22T11:13:48Z)
• LaV-CoT: Language-Aware Visual CoT with Multi-Aspect Reward Optimization for Real-World Multilingual VQA [39.131225916852834]
  Chain-of-thought (CoT) reasoning has been proven to enhance interpretability and complex reasoning.<n>LaV-CoT is the first Language-aware Visual CoT framework with Multi-Aspect Reward Optimization.<n>LaV-CoT achieves up to 9.5% accuracy improvements over open-source baselines.
  arXiv  Detail & Related papers  (2025-09-12T07:45:44Z)
• Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimization [69.29207684569695]
  Chain-of-thought (CoT) reasoning greatly improves the interpretability and problem-solving abilities of multimodal large language models (MLLMs)<n>Existing approaches are focused on text CoT, limiting their ability to leverage visual cues.<n>In this paper, we introduce Unsupervised Visual CoT (UV-CoT), a novel framework for image-level CoT reasoning via preference optimization.
  arXiv  Detail & Related papers  (2025-04-25T14:48:18Z)
• LATTE: Learning to Think with Vision Specialists [110.43838069105998]
  We propose LATTE, a family of vision-language models that offload perception to state-of-the-art vision models.<n>By offloading perception to state-of-the-art vision models, our approach enables vision-language models to focus solely on reasoning over high-quality perceptual information.
  arXiv  Detail & Related papers  (2024-12-07T00:42:04Z)
• LLaVA-CoT: Let Vision Language Models Reason Step-by-Step [34.32147663809707]
  We introduce LLaVA-CoT, a large Vision-Language Model (VLM) designed to conduct autonomous multistage reasoning.<n>Unlike chain-of-thought prompting, LLaVA-CoT independently engages in sequential stages of summarization, visual interpretation, logical reasoning, and conclusion generation.<n>With only 100k training samples and test-time scaling, LLaVA-CoT not only outperforms its base model by 9.4% on a wide range of multimodal reasoning benchmarks.
  arXiv  Detail & Related papers  (2024-11-15T18:58:31Z)
• Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning [40.972648044298374]
  Multi-Modal Large Language Models (MLLMs) have demonstrated impressive performance in various VQA tasks. They often lack interpretability and struggle with complex visual inputs. We introduce the large-scale Visual CoT dataset comprising 438k question-answer pairs. We propose a multi-turn processing pipeline that dynamically focuses on visual inputs and provides interpretable thoughts.
  arXiv  Detail & Related papers  (2024-03-25T17:59:23Z)
• ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting [124.69672273754144]
  Chain-of-Thought (CoT) prompting can enhance the reasoning capabilities of large language models (LLMs) Existing CoT approaches usually focus on simpler reasoning tasks and thus result in low-quality and inconsistent CoT prompts. We introduce CoTGenius, a novel framework designed for the automatic generation of superior CoT prompts.
  arXiv  Detail & Related papers  (2024-03-21T11:34:26Z)
• NPHardEval4V: Dynamic Evaluation of Large Vision-Language Models with Effects of Vision [64.83085920775316]
  We introduce NPHardEval4V, a multimodal benchmark suite grounded in four classical NP-hard problems.<n>Each task is presented through a combination of structured visual layouts and textual prompts, designed to assess the ability of LVLMs to perform reasoning under visual-linguistic constraints.<n>Our results show that while these models perform reasonably well on perception-based inputs, they struggle with global optimization, abstraction, and constraint satisfaction.
  arXiv  Detail & Related papers  (2024-03-04T07:10:31Z)
• Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Texts [65.84370471189676]
  We look at large-scale intermediate pre-training of decomposition-based transformers using distant supervision from comparable texts. We show that with such intermediate pre-training, developing robust decomposition-based models for a diverse range of tasks becomes more feasible.
  arXiv  Detail & Related papers  (2022-10-30T15:38:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.