Bench-CoE: a Framework for Collaboration of Experts from Benchmark

• URL: http://arxiv.org/abs/2412.04167v1
• Date: Thu, 05 Dec 2024 14:03:41 GMT
• Title: Bench-CoE: a Framework for Collaboration of Experts from Benchmark
• Abstract summary: Large Language Models (LLMs) are key technologies driving intelligent systems to handle multiple tasks.<n>This paper proposes the Bench-CoE framework, which enables Collaboration of Experts (CoE) by effectively leveraging benchmark evaluations.
• Score: 6.278928823352044
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are key technologies driving intelligent systems to handle multiple tasks. To meet the demands of various tasks, an increasing number of LLMs-driven experts with diverse capabilities have been developed, accompanied by corresponding benchmarks to evaluate their performance. This paper proposes the Bench-CoE framework, which enables Collaboration of Experts (CoE) by effectively leveraging benchmark evaluations to achieve optimal performance across various tasks. Bench-CoE includes a set of expert models, a router for assigning tasks to corresponding experts, and a benchmark dataset for training the router. Moreover, we formulate Query-Level and Subject-Level approaches based on our framework, and analyze the merits and drawbacks of these two approaches. Finally, we conduct a series of experiments with vary data distributions on both language and multimodal tasks to validate that our proposed Bench-CoE outperforms any single model in terms of overall performance. We hope this method serves as a baseline for further research in this area. The code is available at \url{https://github.com/ZhangXJ199/Bench-CoE}.

Related papers

• UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks [43.36285223987099]
  We introduce UniClawBench, the first capability-driven benchmark to evaluate proactive agents in dynamic, real-world settings.<n>UniClawBench is built around five executor model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination.<n>Unlike previous benchmarks that rely on static, pre-recorded answers, our benchmark evaluates agents in live Docker containers using fine-grained, step-by-step completion checkpoints.
  arXiv  Detail & Related papers  (2026-07-09T17:59:32Z)
• AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction [1.1841228164483055]
  The AEC-Bench is a benchmark for evaluating agentic systems on real-world tasks.<n>This report describes the benchmark motivation, dataset taxonomy, evaluation protocol, and baseline results.
  arXiv  Detail & Related papers  (2026-03-31T03:10:28Z)
• APEX-Searcher: Augmenting LLMs' Search Capabilities through Agentic Planning and Execution [10.643859464982071]
  APEX-Searcher is a novel Agentic Planning and Execution framework to augment large language models (LLMs) search capabilities.<n>It first employs RL with decomposition-specific rewards to optimize strategic planning.<n>It then applies supervised fine-tuning on high-quality multi-hop trajectories to equip the model with robust iterative sub-task execution capabilities.
  arXiv  Detail & Related papers  (2026-03-14T09:17:12Z)
• Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddings [44.77164359074224]
  Multimodal Large Language Models (MLLMs) have become pivotal for advancing Universal Multimodal Embeddings (UME)<n>Recent studies demonstrate that incorporating generative Chain-of-Thought (CoT) reasoning can substantially enhance task-specific representations.<n>We propose a reasoning-driven UME framework that integrates Embedder-Guided Reinforcement Learning (EG-RL) to optimize the Reasoner to produce evidential Traceability CoT.
  arXiv  Detail & Related papers  (2026-02-14T15:35:03Z)
• Token-Level LLM Collaboration via FusionRoute [60.72307345997823]
  FusionRoute is a token-level multi-LLM collaboration framework.<n>It selects the most suitable expert at each decoding step and contributes a complementary logit that refines or corrects the selected expert's next-token distribution.<n>It outperforms both sequence- and token-level collaboration, model merging, and direct fine-tuning.
  arXiv  Detail & Related papers  (2026-01-08T16:53:16Z)
• DocR1: Evidence Page-Guided GRPO for Multi-Page Document Understanding [100.29587871213624]
  We introduce DocR1, an MLLM trained with a novel RL framework, Evidence Page-Guided GRPO.<n>EviGRPO incorporates an evidence-aware reward mechanism that promotes a coarse-to-fine reasoning strategy.<n>We show that DocR1 achieves state-of-the-art performance on multi-page tasks, while consistently maintaining strong results on single-page benchmarks.
  arXiv  Detail & Related papers  (2025-08-10T12:03:45Z)
• Advancing Visual Large Language Model for Multi-granular Versatile Perception [31.78788398688894]
  We present MVP-LM, a Multi-granular and Versatile Perception framework incorporating Visual Large Language Model.<n>Our framework is designed to integrate both word-based and sentence-based perception tasks alongside box and mask predictions.<n>MVP-LM features an innovative multi-granularity decoder in conjunction with a CoT-inspired dataset unification strategy.
  arXiv  Detail & Related papers  (2025-07-22T04:09:14Z)
• CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation [19.071855537400463]
  Large language models (LLMs) play a crucial role in software engineering, excelling in tasks like code generation and maintenance. CoCo-Bench is designed to evaluate LLMs across four critical dimensions: code understanding, code generation, code modification, and code review.
  arXiv  Detail & Related papers  (2025-04-29T11:57:23Z)
• Unveiling Hidden Collaboration within Mixture-of-Experts in Large Language Models [5.211806751260724]
  We propose a hierarchical sparse dictionary learning (HSDL) method that uncovers the collaboration patterns among experts. We also introduce the Contribution-Aware Expert Pruning (CAEP) algorithm, which effectively prunes low-contribution experts.
  arXiv  Detail & Related papers  (2025-04-16T04:06:15Z)
• Reinforcing Compositional Retrieval: Retrieving Step-by-Step for Composing Informative Contexts [67.67746334493302]
  Large Language Models (LLMs) have demonstrated remarkable capabilities across numerous tasks, yet they often rely on external context to handle complex tasks. We propose a tri-encoder sequential retriever that models this process as a Markov Decision Process (MDP) We show that our method consistently and significantly outperforms baselines, underscoring the importance of explicitly modeling inter-example dependencies.
  arXiv  Detail & Related papers  (2025-04-15T17:35:56Z)
• Symbolic Mixture-of-Experts: Adaptive Skill-based Routing for Heterogeneous Reasoning [76.10639521319382]
  We propose Symbolic-MoE, a symbolic, text-based, and gradient-free Mixture-of-Experts framework. We show that Symbolic-MoE's instance-level expert selection improves performance by a large margin but -- when implemented naively -- can introduce a high computational overhead.
  arXiv  Detail & Related papers  (2025-03-07T18:03:13Z)
• P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs [84.24644520272835]
  Large language models (LLMs) showcase varied multilingual capabilities across tasks like translation, code generation, and reasoning. Previous assessments often limited their scope to fundamental natural language processing (NLP) or isolated capability-specific tasks. We present a pipeline for selecting available and reasonable benchmarks from massive ones, addressing the oversight in previous work regarding the utility of these benchmarks. We introduce P-MMEval, a large-scale benchmark covering effective fundamental and capability-specialized datasets.
  arXiv  Detail & Related papers  (2024-11-14T01:29:36Z)
• On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions [29.130355774088205]
  Hierarchical Mixture of Experts (HMoE) excels at handling complex inputs and improving performance on targeted tasks. We theoretically demonstrate that applying tailored gating functions to each expert group allows HMoE to achieve robust results. This includes large-scale multimodal tasks, image classification, and latent domain discovery and prediction tasks, where our modified HMoE models show great performance improvements.
  arXiv  Detail & Related papers  (2024-10-03T19:28:52Z)
• A Large-Scale Evaluation of Speech Foundation Models [110.95827399522204]
  We establish the Speech processing Universal PERformance Benchmark (SUPERB) to study the effectiveness of the foundation model paradigm for speech. We propose a unified multi-tasking framework to address speech processing tasks in SUPERB using a frozen foundation model followed by task-specialized, lightweight prediction heads.
  arXiv  Detail & Related papers  (2024-04-15T00:03:16Z)
• Harder Tasks Need More Experts: Dynamic Routing in MoE Models [58.18526590138739]
  We introduce a novel dynamic expert selection framework for Mixture of Experts (MoE) models. Our method dynamically selects experts based on the confidence level in expert selection for each input.
  arXiv  Detail & Related papers  (2024-03-12T13:41:15Z)
• Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech [107.81472531864195]
  Text language models have shown remarkable zero-shot capability in generalizing to unseen tasks when provided with well-formulated instructions. We present Dynamic-SUPERB, a benchmark for building universal speech models capable of leveraging instruction tuning to perform multiple tasks in a zero-shot fashion.
  arXiv  Detail & Related papers  (2023-09-18T06:43:30Z)
• MultiTool-CoT: GPT-3 Can Use Multiple External Tools with Chain of Thought Prompting [23.607534241574346]
  We propose MultiTool-CoT, a framework that incorporates external tools, such as a calculator and a knowledge retriever, during the reasoning process. We apply MultiTool-CoT to the Task 2 dataset of NumGLUE, which requires both numerical reasoning and domain-specific knowledge.
  arXiv  Detail & Related papers  (2023-05-26T13:00:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.