Diverse LLMs or Diverse Question Interpretations? That is the Ensembling Question

• URL: http://arxiv.org/abs/2507.21168v1
• Date: Fri, 25 Jul 2025 15:26:18 GMT
• Title: Diverse LLMs or Diverse Question Interpretations? That is the Ensembling Question
• Abstract summary: We compare two diversity approaches for answering binary questions using large language models (LLMs)<n>For both cases, we apply majority voting as the ensemble consensus to determine the final answer.<n>Experiments on boolq, strategyqa, and pubmedqa show that question interpretation diversity consistently leads to better ensemble accuracy.
• Score: 5.847084649531299
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Effectively leveraging diversity has been shown to improve performance for various machine learning models, including large language models (LLMs). However, determining the most effective way of using diversity remains a challenge. In this work, we compare two diversity approaches for answering binary questions using LLMs: model diversity, which relies on multiple models answering the same question, and question interpretation diversity, which relies on using the same model to answer the same question framed in different ways. For both cases, we apply majority voting as the ensemble consensus heuristic to determine the final answer. Our experiments on boolq, strategyqa, and pubmedqa show that question interpretation diversity consistently leads to better ensemble accuracy compared to model diversity. Furthermore, our analysis of GPT and LLaMa shows that model diversity typically produces results between the best and the worst ensemble members without clear improvement.

Related papers

• More Is Not More: What Matters for Diversity in LLM Opinions? [0.0]
  We design a factorial experiment that separates two primary intervention dimensions: input conditioning and interaction architecture.<n>We evaluate all conditions on 100 real-user open-ended questions across 7 models, measuring diversity with multiple complementary metrics.<n>Our work demonstrates that diversity is not a product of scaling along any single dimension, but is highly sensitive to the structural form and combination of interventions.
  arXiv  Detail & Related papers  (2026-05-10T00:52:25Z)
• Evaluating the Diversity and Quality of LLM Generated Content [72.84945252821908]
  We introduce a framework for measuring effective semantic diversity--diversity among outputs that meet quality thresholds.<n>Although preference-tuned models exhibit reduced lexical and syntactic diversity, they produce greater effective semantic diversity than SFT or base models.<n>These findings have important implications for applications that require diverse yet high-quality outputs.
  arXiv  Detail & Related papers  (2025-04-16T23:02:23Z)
• Multidimensional Consistency Improves Reasoning in Language Models [21.989335720239467]
  We introduce a framework for testing models for answer consistency across multiple input variations.<n>We induce variations in (i) order of shots in prompt, (ii) problem phrasing, and (iii) languages used.<n>Our framework consistently enhances mathematical reasoning performance on both monolingual dataset GSM8K and multilingual dataset MGSM, especially for smaller models.
  arXiv  Detail & Related papers  (2025-03-04T14:41:05Z)
• DiverseAgentEntropy: Quantifying Black-Box LLM Uncertainty through Diverse Perspectives and Multi-Agent Interaction [53.803276766404494]
  Existing methods, which gauge a model's uncertainty through evaluating self-consistency in responses to the original query, do not always capture true uncertainty.<n>We propose a novel method, DiverseAgentEntropy, for evaluating a model's uncertainty using multi-agent interaction.<n>Our method offers a more accurate prediction of the model's reliability and further detects hallucinations, outperforming other self-consistency-based methods.
  arXiv  Detail & Related papers  (2024-12-12T18:52:40Z)
• Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment [84.32768080422349]
  Alignment with human preference prevents large language models from generating misleading or toxic content. We propose a new formulation of prompt diversity, implying a linear correlation with the final performance of LLMs after fine-tuning.
  arXiv  Detail & Related papers  (2024-03-17T07:08:55Z)
• Diversify Question Generation with Retrieval-Augmented Style Transfer [68.00794669873196]
  We propose RAST, a framework for Retrieval-Augmented Style Transfer. The objective is to utilize the style of diverse templates for question generation. We develop a novel Reinforcement Learning (RL) based approach that maximizes a weighted combination of diversity reward and consistency reward.
  arXiv  Detail & Related papers  (2023-10-23T02:27:31Z)
• Diversifying Question Generation over Knowledge Base via External Natural Questions [17.13395598338518]
  We argue that diverse texts should convey the same semantics through varied expressions.<n>Current metrics inadequately assess the above diversity since they calculate the ratio of unique n-grams in the generated question itself.<n>We devise a new diversity evaluation metric, which measures the diversity among top-k generated questions for each instance.
  arXiv  Detail & Related papers  (2023-09-23T10:37:57Z)
• Getting MoRE out of Mixture of Language Model Reasoning Experts [71.61176122960464]
  We propose a Mixture-of-Reasoning-Experts (MoRE) framework that ensembles diverse specialized language models. We specialize the backbone language model with prompts optimized for different reasoning categories, including factual, multihop, mathematical, and commonsense reasoning. Our human study confirms that presenting expert predictions and the answer selection process helps annotators more accurately calibrate when to trust the system's output.
  arXiv  Detail & Related papers  (2023-05-24T02:00:51Z)
• Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs [78.31625291513589]
  We argue that self-consistency is an important criteria for valid multi-step reasoning in tasks where the solution is composed of the answers to multiple sub-steps. We propose two types of self-consistency that are particularly important for multi-step reasoning -- hypothetical consistency and compositional consistency. We demonstrate that multiple variants of the GPT-3/-4 models exhibit poor consistency rates across both types of consistency on a variety of tasks.
  arXiv  Detail & Related papers  (2023-05-23T17:25:59Z)
• Interpretable Diversity Analysis: Visualizing Feature Representations In Low-Cost Ensembles [0.0]
  This paper introduces several interpretability methods that can be used to qualitatively analyze diversity. We demonstrate these techniques by comparing the diversity of feature representations between child networks using two low-cost ensemble algorithms.
  arXiv  Detail & Related papers  (2023-02-12T00:32:03Z)
• A Unified Theory of Diversity in Ensemble Learning [4.773356856466191]
  We present a theory of ensemble diversity, explaining the nature of diversity for a wide range of supervised learning scenarios. This challenge has been referred to as the holy grail of ensemble learning, an open research issue for over 30 years.
  arXiv  Detail & Related papers  (2023-01-10T13:51:07Z)
• Machine Learning for Multi-Output Regression: When should a holistic multivariate approach be preferred over separate univariate ones? [62.997667081978825]
  Tree-based ensembles such as the Random Forest are modern classics among statistical learning methods. We compare these methods in extensive simulations to help in answering the primary question when to use multivariate ensemble techniques.
  arXiv  Detail & Related papers  (2022-01-14T08:44:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.