Mixture of Complementary Agents for Robust LLM Ensemble

• URL: http://arxiv.org/abs/2605.24048v1
• Date: Thu, 21 May 2026 21:29:05 GMT
• Title: Mixture of Complementary Agents for Robust LLM Ensemble
• Abstract summary: Multi-AI collaboration, such as ensembling or debating large language models (LLMs), is a promising paradigm for aggregating information and boosting performance.<n>Existing approaches primarily focus on accuracy (picking the strongest models) or diversity (ensuring variety)<n>We reframe proposer selection as a selection problem akin to feature selection, where the value of an LLM lies in its complementarity with others.
• Score: 28.210955621253863
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-AI collaboration, such as ensembling or debating large language models (LLMs), is a promising paradigm for aggregating information and boosting performance. A foundational step in these pipelines is to feed the responses of several proposer LLMs into a summarizer LLM, which synthesizes a better answer. However, choosing which proposers to include is non-trivial. Existing approaches primarily focus either on accuracy (picking the strongest models) or diversity (ensuring variety), and often overlook the interactions among proposers and with the summarizer. We reframe proposer selection as a combinatorial selection problem akin to feature selection, where the value of an LLM lies in its complementarity with others. However, directly applying standard feature-selection algorithms is impractical in the LLM setting due to prohibitive time complexity. Motivated by this limitation, we explore an extensive range of computationally feasible, greedy-style selection algorithms that assess complementarity using a small labeled set. Our experiments validate complementarity as a guiding principle for proposer selection and identify methods that achieve the best performance-cost trade-offs in practice.

Related papers

• Data Efficient Sample Selection for In-Context Learning [7.910984819642885]
  In-context learning (ICL) aids large language models (LLMs) to adapt to new tasks without need for fine-tuning.<n>Existing approaches for selection do not model the complex relationship between ICL samples and downstream LLM performance.<n>We introduce DearICL, a new framework that models demonstration example selection as a subset ranking problem.
  arXiv  Detail & Related papers  (2026-09-06T15:23:37Z)
• LISTEN to Your Preferences: An LLM Framework for Multi-Objective Selection [8.355352653283516]
  We introduce LISTEN, a framework that leverages a Large Language Model (LLM) as a zero-shot preference oracle.<n>We show LISTEN-U excels when preferences are parametrically aligned, while LISTEN-T offers more robust performance.<n>This work explores a promising direction for steering complex multi-objective decisions directly with natural language.
  arXiv  Detail & Related papers  (2025-10-29T03:17:37Z)
• Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information [57.397381631496906]
  We develop two new aggregation algorithms called Optimal Weight (OW) and Inverse Surprising Popularity (ISP)<n>Our theoretical analysis shows these methods provably mitigate inherent limitations of majority voting under mild assumptions.<n>We empirically validate our algorithms on synthetic datasets, popular LLM fine-tuning benchmarks such as UltraFeedback and MMLU, and a real-world healthcare setting ARMMAN.
  arXiv  Detail & Related papers  (2025-10-01T22:21:50Z)
• Rethinking LLM-Based Recommendations: A Personalized Query-Driven Parallel Integration [22.650609670923732]
  We propose a parallel recommendation framework that decouples large language models from candidate pre-selection.<n>Our framework connects LLMs and recommendation models in a parallel manner, allowing each component to independently utilize its strengths.
  arXiv  Detail & Related papers  (2025-04-16T09:17:45Z)
• Collab: Controlled Decoding using Mixture of Agents for LLM Alignment [90.6117569025754]
  Reinforcement learning from human feedback has emerged as an effective technique to align Large Language models.<n>Controlled Decoding provides a mechanism for aligning a model at inference time without retraining.<n>We propose a mixture of agent-based decoding strategies leveraging the existing off-the-shelf aligned LLM policies.
  arXiv  Detail & Related papers  (2025-03-27T17:34:25Z)
• Optimizing Knowledge Integration in Retrieval-Augmented Generation with Self-Selection [72.92366526004464]
  Retrieval-Augmented Generation (RAG) has proven effective in enabling Large Language Models (LLMs) to produce more accurate and reliable responses.<n>We propose a novel Self-Selection RAG framework, where the LLM is made to select from pairwise responses generated with internal parametric knowledge solely.
  arXiv  Detail & Related papers  (2025-02-10T04:29:36Z)
• Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes [50.544186914115045]
  Large language models (LLMs) are increasingly embedded in everyday applications.<n> Ensuring their alignment with the diverse preferences of individual users has become a critical challenge.<n>We present a novel framework for few-shot steerable alignment.
  arXiv  Detail & Related papers  (2024-12-18T16:14:59Z)
• A Systematic Examination of Preference Learning through the Lens of Instruction-Following [83.71180850955679]
  We use a novel synthetic data generation pipeline to generate 48,000 instruction unique-following prompts.<n>With our synthetic prompts, we use two preference dataset curation methods - rejection sampling (RS) and Monte Carlo Tree Search (MCTS)<n>Experiments reveal that shared prefixes in preference pairs, as generated by MCTS, provide marginal but consistent improvements.<n>High-contrast preference pairs generally outperform low-contrast pairs; however, combining both often yields the best performance.
  arXiv  Detail & Related papers  (2024-12-18T15:38:39Z)
• SelectLLM: Query-Aware Efficient Selection Algorithm for Large Language Models [8.558834738072363]
  Large language models (LLMs) have been widely adopted due to their remarkable performance across various applications.<n>These individual LLMs show limitations in generalization and performance on complex tasks due to inherent training biases, model size constraints, and the quality or diversity of pre-training datasets.<n>We introduce SelectLLM, which efficiently directs input queries to the most suitable subset of LLMs from a large pool.
  arXiv  Detail & Related papers  (2024-08-16T06:11:21Z)
• Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity [59.57065228857247]
  Retrieval-augmented Large Language Models (LLMs) have emerged as a promising approach to enhancing response accuracy in several tasks, such as Question-Answering (QA) We propose a novel adaptive QA framework, that can dynamically select the most suitable strategy for (retrieval-augmented) LLMs based on the query complexity. We validate our model on a set of open-domain QA datasets, covering multiple query complexities, and show that ours enhances the overall efficiency and accuracy of QA systems.
  arXiv  Detail & Related papers  (2024-03-21T13:52:30Z)
• Federated Neuro-Symbolic Learning [39.04545654772026]
  We present Federated Neuro-Symbolic Learning framework (FedNSL) with latent variables as the FL communication medium. FedNSL is capable of identifying and addressing rule distribution through a simple and effective Kullback-Leibler (KL) divergence constraint. Extensive experiments based on both synthetic and real-world data demonstrate significant advantages of FedNSL compared to five state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-29T14:20:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.