MM-PoE: Multiple Choice Reasoning via. Process of Elimination using Multi-Modal Models

• URL: http://arxiv.org/abs/2412.07148v1
• Date: Tue, 10 Dec 2024 03:13:41 GMT
• Title: MM-PoE: Multiple Choice Reasoning via. Process of Elimination using Multi-Modal Models
• Authors: Sayak Chakrabarty, Souradip Pal,
• Abstract summary: This paper introduces Multiple Choice Reasoning via. Process of Elimination using Multi-Modal models. The novel methodology is engineered to augment the efficacy of Vision-Language Models (VLMs) in multiple-choice visual reasoning tasks. Our empirical evaluations, conducted across three benchmark datasets, reveal that MM-PoE significantly improves both zero-shot and few-shot performance.
• Score: 0.0
• License:
• Abstract: This paper introduces Multiple Choice Reasoning via. Process of Elimination using Multi-Modal models, herein referred to as Multi-Modal Process of Elimination (MM-PoE). This novel methodology is engineered to augment the efficacy of Vision-Language Models (VLMs) in multiple-choice visual reasoning tasks. Diverging from conventional approaches that evaluate each option independently, MM-PoE employs a dual-step scoring paradigm that initially identifies and excludes implausible choices, subsequently concentrating on the most probable remaining options. This method emulates human test-taking strategies, where individuals typically eliminate clearly incorrect answers prior to selecting the optimal response. Our empirical evaluations, conducted across three benchmark datasets, reveal that MM-PoE significantly improves both zero-shot and few-shot performance of contemporary state-of-the-art VLMs. Critically, this approach not only broadens the application of the elimination process to multi-modal contexts but also allows few-shot experiments, thereby addressing two principal limitations concerning usage of PoE only in zero-shot settings and only with a language-only framework. As a result, MM-PoE not only refines the reasoning capabilities of VLMs but also broadens their applicability to complex visual question-answering scenarios. All code and documentation supporting our work are available at https://pypi.org/project/mm-poe/, enabling researchers and practitioners to easily integrate and further develop these techniques.

Related papers

• CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs [107.21334626890713]
  Multimodal Large Language Models (MLLMs) still struggle with hallucinations despite their impressive capabilities. We propose a Cross-modal Hierarchical Direct Preference Optimization (CHiP) to address these limitations. We evaluate CHiP through both quantitative and qualitative analyses, with results across multiple benchmarks demonstrating its effectiveness in reducing hallucinations.
  arXiv  Detail & Related papers  (2025-01-28T02:05:38Z)
• Option-ID Based Elimination For Multiple Choice Questions [12.30777266124562]
  Multiple choice questions (MCQs) are a popular and important task for evaluating large language models (LLMs) Based on common strategies people use when answering MCQs, the process of elimination (PoE) has been proposed as an effective problem-solving method. This paper proposes a PoE based on option ID. Specifically, our method eliminates option by selecting the option ID with the lowest probability.
  arXiv  Detail & Related papers  (2025-01-25T11:06:37Z)
• M$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning [90.75075886543404]
  Multimodal Large Language Models (MLLMs) demonstrate remarkable performance across a wide range of domains. In this work, we introduce a novel Multimodal Prompt Tuning (M$2$PT) approach for efficient instruction tuning of MLLMs.
  arXiv  Detail & Related papers  (2024-09-24T01:40:24Z)
• MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark [77.93283927871758]
  This paper introduces MMMU-Pro, a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning benchmark. MMMU-Pro rigorously assesses multimodal models' true understanding and reasoning capabilities.
  arXiv  Detail & Related papers  (2024-09-04T15:31:26Z)
• NoteLLM-2: Multimodal Large Representation Models for Recommendation [71.87790090964734]
  Large Language Models (LLMs) have demonstrated exceptional proficiency in text understanding and embedding tasks. Their potential in multimodal representation, particularly for item-to-item (I2I) recommendations, remains underexplored. We propose an end-to-end fine-tuning method that customizes the integration of any existing LLMs and vision encoders for efficient multimodal representation.
  arXiv  Detail & Related papers  (2024-05-27T03:24:01Z)
• Plan of Thoughts: Heuristic-Guided Problem Solving with Large Language Models [0.0]
  We formalize a planning-based approach to perform multi-step problem solving with language models. We demonstrate a superior success rate of 89.4% on the Game of 24 task as compared to existing approaches.
  arXiv  Detail & Related papers  (2024-04-29T18:51:17Z)
• Towards Generalist Prompting for Large Language Models by Mental Models [105.03747314550591]
  Large language models (LLMs) have demonstrated impressive performance on many tasks. To achieve optimal performance, specially designed prompting methods are still needed. We introduce the concept of generalist prompting, which operates on the design principle of achieving optimal or near-optimal performance.
  arXiv  Detail & Related papers  (2024-02-28T11:29:09Z)
• POE: Process of Elimination for Multiple Choice Reasoning [19.65826015840337]
  We argue a similar two-step strategy can make LMs better at multiple choice reasoning tasks. In the first step, POE scores each option, and eliminates seemingly wrong options. In the second step, POE masks these wrong options, and makes the final prediction from the remaining options.
  arXiv  Detail & Related papers  (2023-10-24T07:38:43Z)
• Towards Robust Multi-Modal Reasoning via Model Selection [7.6621866737827045]
  LLM serves as the "brain" of the agent, orchestrating multiple tools for collaborative multi-step task solving. We propose the $textitM3$ framework as a plug-in with negligible runtime overhead at test-time. Our experiments reveal that our framework enables dynamic model selection, considering both user inputs and subtask dependencies.
  arXiv  Detail & Related papers  (2023-10-12T16:06:18Z)
• KECP: Knowledge Enhanced Contrastive Prompting for Few-shot Extractive Question Answering [28.18555591429343]
  We propose a novel framework named Knowledge Enhanced Contrastive Prompt-tuning (KECP) Instead of adding pointer heads to PLMs, we transform the task into a non-autoregressive Masked Language Modeling (MLM) generation problem. Our method consistently outperforms state-of-the-art approaches in few-shot settings by a large margin.
  arXiv  Detail & Related papers  (2022-05-06T08:31:02Z)
• DUMA: Reading Comprehension with Transposition Thinking [107.89721765056281]
  Multi-choice Machine Reading (MRC) requires model to decide the correct answer from a set of answer options when given a passage and a question. New DUal Multi-head Co-Attention (DUMA) model is inspired by human's transposition thinking process solving the multi-choice MRC problem.
  arXiv  Detail & Related papers  (2020-01-26T07:35:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.