OmniDiagram: Advancing Unified Diagram Code Generation via Visual Interrogation Reward

• URL: http://arxiv.org/abs/2604.05514v1
• Date: Tue, 07 Apr 2026 07:10:24 GMT
• Title: OmniDiagram: Advancing Unified Diagram Code Generation via Visual Interrogation Reward
• Abstract summary: We introduce a novel visual feedback strategy named Visual Interrogation Verifies All (textscViva)<n>textscViva rewards the visual structure of rendered diagrams through a generative approach.<n>We construct M3$2$Diagram, the first large-scale diagram code generation dataset.
• Score: 8.227370271724162
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The paradigm of programmable diagram generation is evolving rapidly, playing a crucial role in structured visualization. However, most existing studies are confined to a narrow range of task formulations and language support, constraining their applicability to diverse diagram types. In this work, we propose OmniDiagram, a unified framework that incorporates diverse diagram code languages and task definitions. To address the challenge of aligning code logic with visual fidelity in Reinforcement Learning (RL), we introduce a novel visual feedback strategy named Visual Interrogation Verifies All (\textsc{Viva}). Unlike brittle syntax-based rules or pixel-level matching, \textsc{Viva} rewards the visual structure of rendered diagrams through a generative approach. Specifically, \textsc{Viva} actively generates targeted visual inquiries to scrutinize diagram visual fidelity and provides fine-grained feedback for optimization. This mechanism facilitates a self-evolving training process, effectively obviating the need for manually annotated ground truth code. Furthermore, we construct M3$^2$Diagram, the first large-scale diagram code generation dataset, containing over 196k high-quality instances. Experimental results confirm that the combination of SFT and our \textsc{Viva}-based RL allows OmniDiagram to establish a new state-of-the-art (SOTA) across diagram code generation benchmarks.

Related papers

• From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding [0.0]
  Vision-language models (VLMs) have demonstrated strong performance in visual question answering with natural images.<n>We introduce a framework for generating large-scale diagram-grounded instruction data by leveraging terminology derived from scientific curricula.<n>We construct SciGram, a dataset of over 194K diagrams and 1.4M visual instructions across life, earth, and physical sciences.
  arXiv  Detail & Related papers  (2026-09-01T09:07:31Z)
• Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification [80.62512020268626]
  UniAR is a unified autoregressive framework where a single visual tokenizer serves as the key bridge between understanding and generation.<n>UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme.<n>A diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images.
  arXiv  Detail & Related papers  (2026-06-16T17:59:22Z)
• VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editing [37.60481878724908]
  We present VCG-Bench, a unified benchmark for visual-centrictext ttmxGraph tasks.<n> VCG-Bench comprises: (1) a taxonomized dataset of 1,449 diverse diagrams spanning 6 domains and 15 sub-domains, (2) a paradigm definition that integrates Generation (Vision-to-Code) and Editability (Code-to-Code), and (3) a Tailored Evaluation Protocol employing multi-dimensional metrics such as textttmxGraph Execution Success Rate, Style Consistency Score (SCS), etc.
  arXiv  Detail & Related papers  (2026-05-15T06:59:28Z)
• MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval [111.99106496142474]
  Visual Document Retrieval (VDR) requires representations that capture both fine-grained visual details and global document structure.<n>Existing VDR models struggle to balance effectiveness and efficiency when processing high-resolution documents.<n>We propose a new X-VisEmb paradigm that progresses from multi-resolution sampling and encoding, through cross-granularity feature fusion, to adaptive representation distillation.
  arXiv  Detail & Related papers  (2026-03-07T15:17:22Z)
• EvoDiagram: Agentic Editable Diagram Creation via Design Expertise Evolution [46.35311453619801]
  EvoDiagram is an agentic framework that generates object-level editable diagrams via an intermediate canvas schema.<n>EvoDiagram employs a coordinated multi-agent system to decouple semantic intent from rendering logic.<n>We further release CanvasBench, a benchmark consisting of both data and metrics for canvas-based diagramming.
  arXiv  Detail & Related papers  (2026-02-20T11:11:02Z)
• Aligning Text, Code, and Vision: A Multi-Objective Reinforcement Learning Framework for Text-to-Visualization [50.13408999553116]
  We propose RL-Text2Vis, the first reinforcement learning framework for Text2Vis generation.<n>Our method uses a novel multi-objective reward that jointly optimize textual accuracy, code validity, and visualization quality.<n>Our results establish GRPO as an effective strategy for structured, multimodal reasoning in visualization generation.
  arXiv  Detail & Related papers  (2026-01-08T04:29:07Z)
• DiagramEval: Evaluating LLM-Generated Diagrams via Graphs [25.040934047462112]
  We argue that a promising direction is to generate demonstration diagrams directly in textual form as SVGs.<n>We propose DiagramEval, a novel evaluation metric designed to assess the quality of demonstration diagrams generated by large language models.
  arXiv  Detail & Related papers  (2025-10-29T17:56:17Z)
• RECODE: Reasoning Through Code Generation for Visual Question Answering [68.86938437188964]
  We propose to leverage derendering -- the process of reverse-engineering visuals into executable code -- as a new modality for verifiable visual reasoning.<n>Our work demonstrates that grounding visual perception in executable code provides a new path toward more accurate and verifiable multimodal reasoning.
  arXiv  Detail & Related papers  (2025-10-15T17:05:37Z)
• SmartCLIP: Modular Vision-language Alignment with Identification Guarantees [59.16312652369709]
  Contrastive Language-Image Pre-training (CLIP)citepradford2021learning has emerged as a pivotal model in computer vision and multimodal learning.<n>CLIP struggles with potential information misalignment in many image-text datasets and suffers from entangled representation.<n>We introduce ours, a novel approach that identifies and aligns the most relevant visual and textual representations in a modular manner.
  arXiv  Detail & Related papers  (2025-07-29T22:26:20Z)
• Draw with Thought: Unleashing Multimodal Reasoning for Scientific Diagram Generation [4.246322465055928]
  We propose Draw with Thought (DwT), a training-free framework that guides MLLMs to reconstruct diagrams into editable mxGraph XML code.<n>DwT enables interpretable and controllable outputs without model fine-tuning.<n>We release Plot2XML, a benchmark of 247 real-world scientific diagrams with gold-standard XML annotations.
  arXiv  Detail & Related papers  (2025-04-13T08:22:09Z)
• Boosting Chart-to-Code Generation in MLLM via Dual Preference-Guided Refinement [16.22363384653305]
  Multimodal Large Language Models (MLLMs) perform fine-grained visual parsing, precise code synthesis, and robust cross-modal reasoning.<n>We propose a dual preference-guided refinement framework that combines a feedback-driven, dual-modality reward mechanism with iterative preference learning.<n>Our framework significantly enhances the performance of general-purpose open-source MLLMs, enabling them to generate high-quality plotting code.
  arXiv  Detail & Related papers  (2025-04-03T07:51:20Z)
• Instruction Tuning-free Visual Token Complement for Multimodal LLMs [51.138806401996696]
  multimodal large language models (MLLMs) have promised an elegant bridge between vision and language. We propose a Visual Token Complement framework (VTC) that helps MLLMs regain the missing visual features. Our VTC integrates text-to-image generation as a guide to identifying the text-irrelevant features, and a visual selector is then developed to generate complementary visual tokens.
  arXiv  Detail & Related papers  (2024-08-09T12:13:01Z)
• Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models [81.71651422951074]
  Chain-of-Spot (CoS) method is a novel approach that enhances feature extraction by focusing on key regions of interest. This technique allows LVLMs to access more detailed visual information without altering the original image resolution. Our empirical findings demonstrate a significant improvement in LVLMs' ability to understand and reason about visual content.
  arXiv  Detail & Related papers  (2024-03-19T17:59:52Z)
• SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning [61.57887011165744]
  multimodal Transformers have made great progress in the task of Visual Commonsense Reasoning. We propose a Scene Graph Enhanced Image-Text Learning framework to incorporate visual scene graphs in commonsense reasoning.
  arXiv  Detail & Related papers  (2021-12-16T03:16:30Z)
• Weakly Supervised Visual Semantic Parsing [49.69377653925448]
  Scene Graph Generation (SGG) aims to extract entities, predicates and their semantic structure from images. Existing SGG methods require millions of manually annotated bounding boxes for training. We propose Visual Semantic Parsing, VSPNet, and graph-based weakly supervised learning framework.
  arXiv  Detail & Related papers  (2020-01-08T03:46:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.