Arrow-Guided VLM: Enhancing Flowchart Understanding via Arrow Direction Encoding

• URL: http://arxiv.org/abs/2505.07864v1
• Date: Fri, 09 May 2025 04:27:36 GMT
• Title: Arrow-Guided VLM: Enhancing Flowchart Understanding via Arrow Direction Encoding
• Abstract summary: Flowcharts are indispensable tools in software design and business-process analysis.<n>Current vision-language models (VLMs) frequently misinterpret the directional arrows and graph topology.<n>We introduce a seven-stage pipeline grouped into three broader processes.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Flowcharts are indispensable tools in software design and business-process analysis, yet current vision-language models (VLMs) frequently misinterpret the directional arrows and graph topology that set these diagrams apart from natural images. We introduce a seven-stage pipeline grouped into three broader processes: (1) arrow-aware detection of nodes and arrow endpoints; (2) optical character recognition (OCR) to extract node text; and (3) construction of a structured prompt that guides the VLMs. Tested on a 90-question benchmark distilled from 30 annotated flowcharts, the method raises overall accuracy from 80 % to 89 % (+9 percentage points) without any task-specific fine-tuning. The gain is most pronounced for next-step queries (25/30 -> 30/30; 100 %, +17 pp); branch-result questions improve more modestly, and before-step questions remain difficult. A parallel evaluation with an LLM-as-a-Judge protocol shows the same trends, reinforcing the advantage of explicit arrow encoding. Limitations include dependence on detector and OCR precision, the small evaluation set, and residual errors at nodes with multiple incoming edges. Future work will enlarge the benchmark with synthetic and handwritten flowcharts and assess the approach on Business Process Model and Notation (BPMN) and Unified Modeling Language (UML).

Related papers

• Visual Information Extraction from Documents via Classification-Guided Large Vision-Language Models [7.176656402971074]
  We propose a classification-guided large vision-language model (LVLM) framework for multi-type VIE that achieves high accuracy with minimal supervision.<n>The framework offers an efficient, scalable solution for complex document understanding in office automation.
  arXiv  Detail & Related papers  (2026-07-22T03:27:50Z)
• EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design [0.4499833362998488]
  We introduce a benchmark suite with three evaluation dimensions.<n>We present EngiAI, a Multi-Agent System (MAS) implementation built on LangGraph.
  arXiv  Detail & Related papers  (2026-05-19T12:12:09Z)
• Towards Improved Sentence Representations using Token Graphs [41.412173502714225]
  We introduce GLOT, a structure-aware pooling module that reframes pooling as relational learning followed by aggregation.<n>On a diagnostic stress test where 90% of tokens are random distractors, GLOT maintains over 97% accuracy while baseline methods collapse.<n>It is competitive with state-of-the-art techniques on benchmarks like GLUE and MTEB with 20x fewer trainable parameters and speeds up the training time by over 100x compared with parameter-efficient fine-tuning methods.
  arXiv  Detail & Related papers  (2026-03-03T09:00:01Z)
• Semi-supervised Instruction Tuning for Large Language Models on Text-Attributed Graphs [62.544129365882014]
  We propose a novel Semi-supervised Instruction Tuning pipeline for Graph Learning, named SIT-Graph.<n> SIT-Graph is model-agnostic and can be seamlessly integrated into any graph instruction tuning method that utilizes LLMs as the predictor.<n>Extensive experiments demonstrate that when incorporated into state-of-the-art graph instruction tuning methods, SIT-Graph significantly enhances their performance on text-attributed graph benchmarks.
  arXiv  Detail & Related papers  (2026-01-19T08:10:53Z)
• Scaling Graph Chain-of-Thought Reasoning: A Multi-Agent Framework with Efficient LLM Serving [38.059017394879284]
  Graph Chain-of-Thought (Graph-CoT) enables large language models (LLMs) to perform step-by-step reasoning over graph-structured knowledge.<n>Existing pipelines suffer from low accuracy, excessive token usage, high latency, and low throughput.<n>We present GLM, the first multi-agent Graph-CoT system co-designed with an optimized LLM serving architecture.
  arXiv  Detail & Related papers  (2025-11-03T14:42:53Z)
• EncQA: Benchmarking Vision-Language Models on Visual Encodings for Charts [13.788477482875855]
  Multimodal vision-language models (VLMs) continue to achieve ever-improving scores on chart understanding benchmarks.<n>We introduce EncQA, a novel benchmark designed to provide systematic coverage of visual encodings and analytic tasks.<n>Our evaluation of 9 state-of-the-art VLMs reveals that performance varies significantly across encodings within the same task, as well as across tasks.
  arXiv  Detail & Related papers  (2025-08-06T17:17:46Z)
• Learning Efficient and Generalizable Graph Retriever for Knowledge-Graph Question Answering [75.12322966980003]
  Large Language Models (LLMs) have shown strong inductive reasoning ability across various domains.<n>Most existing RAG pipelines rely on unstructured text, limiting interpretability and structured reasoning.<n>Recent studies have explored integrating knowledge graphs with LLMs for knowledge graph question answering.<n>We propose RAPL, a novel framework for efficient and effective graph retrieval in KGQA.
  arXiv  Detail & Related papers  (2025-06-11T12:03:52Z)
• Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents [106.04963073116468]
  Flowcharts are a critical tool for visualizing decision-making processes.<n> vision-language models frequently hallucinate nonexistent connections and decision paths when analyzing these diagrams.<n>We introduce Fine-grained Flowchart, which traces specific components grounding a flowchart referring LLM response.<n>We propose FlowPathAgent, a neurosymbolic agent that performs fine-grained post hoc attribution through graph-based reasoning.
  arXiv  Detail & Related papers  (2025-06-02T06:02:41Z)
• Socratic Chart: Cooperating Multiple Agents for Robust SVG Chart Understanding [14.75820681491341]
  Existing benchmarks reveal reliance on text-based shortcuts and probabilistic pattern-matching rather than genuine visual reasoning.<n>We propose Socratic Chart, a new framework that transforms chart images into Scalable Vector Graphics representations.<n>Our framework surpasses state-of-the-art models in accurately capturing chart primitives and improving reasoning performance.
  arXiv  Detail & Related papers  (2025-04-14T00:07:39Z)
• Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Framework [17.838177710655287]
  Multimodal Retrieval-Augmented Generation (MRAG) enhances reasoning capabilities by integrating external knowledge.<n>Existing benchmarks primarily focus on simple image-text interactions, overlooking complex visual formats like charts that are prevalent in real-world applications.<n>We propose CHARt-based document question-answering GEneration (CHARGE), a framework that produces evaluation data through structured keypoint extraction, crossmodal verification, and keypoint-based generation.
  arXiv  Detail & Related papers  (2025-02-20T18:59:42Z)
• Enhancing Input-Label Mapping in In-Context Learning with Contrastive Decoding [71.01099784480597]
  Large language models (LLMs) excel at a range of tasks through in-context learning (ICL)<n>We introduce In-Context Contrastive Decoding (ICCD), a novel method that emphasizes input-label mapping.<n>ICCD emphasizes input-label mapping by contrasting the output distributions between positive and negative in-context examples.
  arXiv  Detail & Related papers  (2025-02-19T14:04:46Z)
• Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understanding [9.267156820352996]
  Flowcharts are typically presented as images, driving the trend of using vision-language models (VLMs) for end-to-end flowchart understanding.<n>Two key challenges arise: (i) Limited controllability--users have minimal influence over the downstream task, while the training of VLMs is often out of reach.<n>We propose TextFlow, addressing aforementioned issues with two stages: Vision Textualizer and Textual Reasoner.
  arXiv  Detail & Related papers  (2024-12-21T00:52:41Z)
• Instance-Aware Graph Prompt Learning [71.26108600288308]
  We introduce Instance-Aware Graph Prompt Learning (IA-GPL) in this paper. The process involves generating intermediate prompts for each instance using a lightweight architecture. Experiments conducted on multiple datasets and settings showcase the superior performance of IA-GPL compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-11-26T18:38:38Z)
• NAMER: Non-Autoregressive Modeling for Handwritten Mathematical Expression Recognition [80.22784377150465]
  Handwritten Mathematical Expression Recognition (HMER) has gained considerable attention in pattern recognition for its diverse applications in document understanding. This paper makes the first attempt to build a novel bottom-up Non-AutoRegressive Modeling approach for HMER, called NAMER. NAMER comprises a Visual Aware Tokenizer (VAT) and a Parallel Graph (PGD)
  arXiv  Detail & Related papers  (2024-07-16T04:52:39Z)
• FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding [52.35520385083425]
  FlowLearn dataset is a resource tailored to enhance the understanding of flowcharts. The scientific subset contains 3,858 flowcharts sourced from scientific literature. The simulated subset contains 10,000 flowcharts created using a customizable script.
  arXiv  Detail & Related papers  (2024-07-06T20:58:51Z)
• Momentum Decoding: Open-ended Text Generation As Graph Exploration [49.812280360794894]
  Open-ended text generation with autoregressive language models (LMs) is one of the core tasks in natural language processing. We formulate open-ended text generation from a new perspective, i.e., we view it as an exploration process within a directed graph. We propose a novel decoding method -- textitmomentum decoding -- which encourages the LM to explore new nodes outside the current graph.
  arXiv  Detail & Related papers  (2022-12-05T11:16:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.