Hierarchical Vision-Language Reasoning for Multimodal Multiple-Choice Question Answering

• URL: http://arxiv.org/abs/2508.16148v1
• Date: Fri, 22 Aug 2025 07:17:16 GMT
• Title: Hierarchical Vision-Language Reasoning for Multimodal Multiple-Choice Question Answering
• Authors: Ao Zhou, Zebo Gu, Tenghao Sun, Jiawen Chen, Mingsheng Tu, Zifeng Cheng, Yafeng Yin, Zhiwei Jiang, Qing Gu,
• Abstract summary: Multimodal Large Language Models (MLLMs) have demonstrated remarkable multimodal understanding capabilities in Visual Question Answering (VQA) tasks.<n>Current mainstream models suffer from a strong bias toward English training data, resulting in suboptimal performance for Japanese and other language scenarios.<n>This paper proposes a novel Japanese PDF document understanding framework that combines multimodal hierarchical reasoning mechanisms with Colqwen-optimized retrieval methods.
• Score: 13.298532858905782
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Large Language Models (MLLMs) have demonstrated remarkable multimodal understanding capabilities in Visual Question Answering (VQA) tasks by integrating visual and textual features. However, under the challenging ten-choice question evaluation paradigm, existing methods still exhibit significant limitations when processing PDF documents with complex layouts and lengthy content. Notably, current mainstream models suffer from a strong bias toward English training data, resulting in suboptimal performance for Japanese and other language scenarios. To address these challenges, this paper proposes a novel Japanese PDF document understanding framework that combines multimodal hierarchical reasoning mechanisms with Colqwen-optimized retrieval methods, while innovatively introducing a semantic verification strategy through sub-question decomposition. Experimental results demonstrate that our framework not only significantly enhances the model's deep semantic parsing capability for complex documents, but also exhibits superior robustness in practical application scenarios.

Related papers

• Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding [49.26132236798123]
  Vision Language Models (VLMs) have gradually become a primary approach in document understanding.<n>We propose SLEUTH, a multi agent framework that orchestrates a retriever and four collaborative agents in a coarse to fine process.<n>The framework identifies key textual and visual clues within the retrieved pages, filters for salient visual evidence such as tables and charts, and analyzes the query to devise a reasoning strategy.
  arXiv  Detail & Related papers  (2025-11-28T03:09:40Z)
• Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding [61.36285696607487]
  Document understanding is critical for applications from financial analysis to scientific discovery.<n>Current approaches, whether OCR-based pipelines feeding Large Language Models (LLMs) or native Multimodal LLMs (MLLMs) face key limitations.<n>Retrieval-Augmented Generation (RAG) helps ground models in external data, but documents' multimodal nature, combining text, tables, charts, and layout, demands a more advanced paradigm: Multimodal RAG.
  arXiv  Detail & Related papers  (2025-10-17T02:33:16Z)
• DocThinker: Explainable Multimodal Large Language Models with Rule-based Reinforcement Learning for Document Understanding [66.07724324530844]
  We propose DocThinker, a rule-based Reinforcement Learning framework for dynamic inference-time reasoning.<n>Our method mitigates catastrophic forgetting and enhances both adaptability and transparency.<n>Our findings highlight RL as a powerful alternative for enhancing explainability and adaptability in MLLM-based document understanding.
  arXiv  Detail & Related papers  (2025-08-12T03:06:55Z)
• Structured Attention Matters to Multimodal LLMs in Document Understanding [52.37530640460363]
  We investigate how input format influences document comprehension performance.<n>We discover that raw OCR text often impairs rather than improves MLLMs' performance.<n>We propose a novel structure-preserving approach that encodes document elements using the LaTex paradigm.
  arXiv  Detail & Related papers  (2025-06-19T07:16:18Z)
• Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering [60.062194349648195]
  Document Visual Question Answering (DocVQA) faces dual challenges in processing lengthy multimodal documents.<n>Current document retrieval-augmented generation (DocRAG) methods remain limited by their text-centric approaches.<n>We introduce MMDocRAG, a comprehensive benchmark featuring 4,055 expert-annotated QA pairs with multi-page, cross-modal evidence chains.
  arXiv  Detail & Related papers  (2025-05-22T09:52:57Z)
• Beyond Retrieval: Joint Supervision and Multimodal Document Ranking for Textbook Question Answering [3.6799953119508735]
  We propose a novel approach to multimodal textbook question answering by introducing a mechanism for enhancing semantic representations.<n>Our model, Joint Embedding Training With Ranking Supervision for Textbook Question Answering (JETRTQA), is a multimodal learning framework built on a retriever-generator architecture.<n>We evaluate our method on the CK12-QA dataset and demonstrate that it significantly improves the discrimination between informative and irrelevant documents.
  arXiv  Detail & Related papers  (2025-05-17T13:23:54Z)
• A Multi-Granularity Retrieval Framework for Visually-Rich Documents [4.804551482123172]
  We propose a unified multi-granularity multimodal retrieval framework tailored for two benchmark tasks: MMDocIR and M2KR.<n>Our approach integrates hierarchical encoding strategies, modality-aware retrieval mechanisms, and vision-language model (VLM)-based candidate filtering.<n>Our framework demonstrates robust performance without the need for task-specific fine-tuning.
  arXiv  Detail & Related papers  (2025-05-01T02:40:30Z)
• Multi2: Multi-Agent Test-Time Scalable Framework for Multi-Document Processing [43.75154489681047]
  We propose a novel framework leveraging test-time scaling for Multi-Document Summarization (MDS)<n>Our approach employs prompt ensemble techniques to generate multiple candidate summaries using various prompts, then combines them with an aggregator to produce a refined summary.<n>To evaluate our method effectively, we also introduce two new LLM-based metrics: the Consistency-Aware Preference (CAP) score and LLM Atom-Content-Unit (LLM-ACU) score.
  arXiv  Detail & Related papers  (2025-02-27T23:34:47Z)
• VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation [100.06122876025063]
  This paper introduces VisDoMBench, the first comprehensive benchmark designed to evaluate QA systems in multi-document settings.<n>We propose VisDoMRAG, a novel multimodal Retrieval Augmented Generation (RAG) approach that simultaneously utilizes visual and textual RAG.
  arXiv  Detail & Related papers  (2024-12-14T06:24:55Z)
• CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Model [9.224965304457708]
  This paper introduces Contextual Understanding and Enhanced Search with MLLM (CUE-M), a novel multimodal search framework.<n>It incorporates image context enrichment, intent refinement, contextual query generation, external API integration, and relevance-based filtering.<n>Experiments on real-word datasets and public benchmarks on knowledge-based VQA and safety demonstrated that CUE-M outperforms baselines and establishes new state-of-the-art results.
  arXiv  Detail & Related papers  (2024-11-19T07:16:48Z)
• SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization [49.931663904599205]
  Researchers have developed techniques to develop Large Multimodal Models with In-Context Learning capabilities. Existing LMMs fail to effectively leverage the visual context in multimodal demonstrations and instead simply follow textual patterns. We propose Symbol Demonstration Direct Preference Optimization (SymDPO) to break the traditional paradigm of constructing multimodal demonstrations.
  arXiv  Detail & Related papers  (2024-11-17T08:29:14Z)
• Multi-modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models [25.058162782167503]
  Chain-of-thought (CoT) reasoning has exhibited impressive performance in language models for solving complex tasks and answering questions. We introduce a novel approach for multi-modal CoT reasoning that utilizes latent space learning via diffusion processes to generate effective image features that align with language thoughts. Our method fuses image features and text representations at a deep level and improves the complex reasoning ability of multi-modal CoT.
  arXiv  Detail & Related papers  (2023-12-14T09:13:09Z)
• OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models [122.27878464009181]
  We conducted a comprehensive evaluation of Large Multimodal Models, such as GPT4V and Gemini, in various text-related visual tasks. OCRBench contains 29 datasets, making it the most comprehensive OCR evaluation benchmark available.
  arXiv  Detail & Related papers  (2023-05-13T11:28:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.