Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QA

• URL: http://arxiv.org/abs/2605.14928v1
• Date: Thu, 14 May 2026 15:03:36 GMT
• Title: Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QA
• Abstract summary: ProcedureVQA is a novel multimodal benchmark specifically designed for visual procedural reasoning.<n>We present Chain-of-Procedure (CoP), a hierarchical reasoning framework that first retrieves relevant instructions using visual cues, then performs step refinement through semantic decomposition, and finally generates the next step.
• Score: 36.09221196975847
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in vision-language models (VLMs) have achieved impressive results on standard image-text tasks, yet their potential for visual procedure question answering (VP-QA) remains largely unexplored. VP-QA presents unique challenges where users query next-step actions by uploading images for intermediate states of complex procedures. To systematically evaluate VLMs on this practical task, we propose ProcedureVQA, a novel multimodal benchmark specifically designed for visual procedural reasoning. Through comprehensive analysis, we identify two critical limitations in current VLMs: inadequate cross-modal retrieval of structured procedures given visual states, and misalignment between image sequence granularity and textual step decomposition. To address these issues, we present Chain-of-Procedure (CoP), a hierarchical reasoning framework that first retrieves relevant instructions using visual cues, then performs step refinement through semantic decomposition, and finally generates the next step. Experiments across six VLMs demonstrate CoP's effectiveness, achieving up to 13% absolute improvement over standard baselines.

Related papers

• MATEO: A Multimodal Benchmark for Temporal Reasoning and Planning in LVLMs [2.1793134762413433]
  AI agents need to plan to achieve complex goals that involve orchestrating perception, sub-goal decomposition, and execution.<n>Existing research on foundational models' understanding of temporal execution is limited to automatically derived annotations, approximations of the TEO as a linear chain, or text-only inputs.<n>We introduce MATEO, a benchmark designed to assess and improve the temporal reasoning abilities of Large Vision Language Models (LVLMs) required for real-world planning.
  arXiv  Detail & Related papers  (2026-02-16T09:41:50Z)
• Learning Procedural-aware Video Representations through State-Grounded Hierarchy Unfolding [12.083424392460406]
  We introduce'states', i.e., textual snapshots of object configurations, as a visually-grounded semantic layer that anchors procedures to what a model can actually see.<n>We formalize this insight in a novel Task-Step-State (TSS) framework, where tasks are achieved via steps that drive transitions between observable states.<n>Our method outperforms baseline models on multiple downstream tasks, including task recognition, step recognition, and next step prediction.
  arXiv  Detail & Related papers  (2025-11-25T08:46:11Z)
• QG-CoC: Question-Guided Chain-of-Captions for Large Multimodal Models [50.51641024244313]
  We investigate how current prompting methods perceive fine-grained visual details and process visual information when dealing with multiple images.<n>Inspired by the findings, we propose a new zero-shot prompting method, Question-Guided Chain-of-Captions (QG-CoC)<n>We evaluate our method on various open-source and closed-source MLLMs for multi-image and single-image benchmarks.
  arXiv  Detail & Related papers  (2025-11-05T05:49:48Z)
• CIR-CoT: Towards Interpretable Composed Image Retrieval via End-to-End Chain-of-Thought Reasoning [93.05917922306196]
  Composed Image Retrieval (CIR) aims to find a target image from a reference image and a modification text.<n>CIR-CoT is the first end-to-end retrieval-oriented MLLM designed to integrate explicit Chain-of-Thought (CoT) reasoning.
  arXiv  Detail & Related papers  (2025-10-09T09:41:45Z)
• ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models [67.75439511654078]
  Large Vision-Language Models (LVLMs) have introduced a new paradigm for understanding and reasoning about image input through textual responses.<n>They face the persistent challenge of hallucination, which introduces practical weaknesses and raises concerns about their reliable deployment in real-world applications.<n>We propose ONLY, a training-free decoding approach that requires only a single query and a one-layer intervention during decoding, enabling efficient real-time deployment.
  arXiv  Detail & Related papers  (2025-07-01T16:01:08Z)
• ImageScope: Unifying Language-Guided Image Retrieval via Large Multimodal Model Collective Reasoning [62.61187785810336]
  ImageScope is a training-free, three-stage framework that unifies language-guided image retrieval tasks.<n>In the first stage, we improve the robustness of the framework by synthesizing search intents across varying levels of semantic granularity.<n>In the second and third stages, we reflect on retrieval results by verifying predicate propositions locally, and performing pairwise evaluations globally.
  arXiv  Detail & Related papers  (2025-03-13T08:43:24Z)
• Multi2: Multi-Agent Test-Time Scalable Framework for Multi-Document Processing [43.75154489681047]
  We propose a novel framework leveraging test-time scaling for Multi-Document Summarization (MDS)<n>Our approach employs prompt ensemble techniques to generate multiple candidate summaries using various prompts, then combines them with an aggregator to produce a refined summary.<n>To evaluate our method effectively, we also introduce two new LLM-based metrics: the Consistency-Aware Preference (CAP) score and LLM Atom-Content-Unit (LLM-ACU) score.
  arXiv  Detail & Related papers  (2025-02-27T23:34:47Z)
• VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning [63.0285363282581]
  Multimodal Large Language Models (MLLMs) have become a powerful tool for integrating visual and textual information.<n>We introduce VOILA, a benchmark designed to evaluate MLLMs' perceptual understanding and abstract relational reasoning.<n>We reveal that current MLLMs struggle to comprehend inter-image relationships and exhibit limited capabilities in high-level relational reasoning.
  arXiv  Detail & Related papers  (2025-02-25T23:36:19Z)
• From Visuals to Vocabulary: Establishing Equivalence Between Image and Text Token Through Autoregressive Pre-training in MLLMs [23.011836329934255]
  Vision Dynamic Embedding-Guided Pretraining (VDEP) is a hybrid autoregressive training paradigm for MLLMs.<n>The proposed method seamlessly integrates into standard models without architectural changes.<n> Experiments on 13 benchmarks show VDEP outperforms baselines, surpassing existing methods.
  arXiv  Detail & Related papers  (2025-02-13T09:04:28Z)
• Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage [50.84150600032693]
  Multimodal large language models (MLLMs) excel at generating highly detailed captions but often produce hallucinations.<n>We propose a multiagent approach that leverages LLM-MLLM collaboration to correct given captions.<n>Our proposed method significantly enhances the factual accuracy of captions, even improving those generated by GPT-4V.
  arXiv  Detail & Related papers  (2024-12-20T01:37:22Z)
• Trust but Verify: Programmatic VLM Evaluation in the Wild [62.14071929143684]
  Programmatic VLM Evaluation (PROVE) is a new benchmarking paradigm for evaluating VLM responses to open-ended queries. We benchmark the helpfulness-truthfulness trade-offs of a range ofVLMs on PROVE, finding that very few are in-fact able to achieve a good balance between the two.
  arXiv  Detail & Related papers  (2024-10-17T01:19:18Z)
• ExoViP: Step-by-step Verification and Exploration with Exoskeleton Modules for Compositional Visual Reasoning [27.725814615823687]
  We propose a "plug-and-play" method, ExoViP, to correct errors in both the planning and execution stages. We employ verification modules as "exoskeletons" to enhance current vision-language programming schemes.
  arXiv  Detail & Related papers  (2024-08-05T03:22:10Z)
• Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering [6.798129852396113]
  In this paper, we explore effective prompting techniques to enhance zero- and few-shot Visual Question Answering (VQA) performance.<n>We identify that specific templates significantly influence VQA outcomes, underscoring the need for strategic template selection.<n>To mitigate the challenges associated with evaluating free-form open-ended VQA responses, we introduce a straightforward LLM-guided pre-processing technique.
  arXiv  Detail & Related papers  (2023-06-16T17:47:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.