Differentiable Mixture-of-Agents Incentivizes Swarm Intelligence of Large Language Models

• URL: http://arxiv.org/abs/2605.15706v1
• Date: Fri, 15 May 2026 07:54:46 GMT
• Title: Differentiable Mixture-of-Agents Incentivizes Swarm Intelligence of Large Language Models
• Abstract summary: Differentiable Mixture-of-Agents (DMoA) is a self-evolving multi-agent framework that enables elastic and adaptive agent collaboration during inference.<n>We show that DMoA achieves state-of-the-art performance while exhibiting strong efficiency, robustness, and ensembling capabilities.
• Score: 17.7409616106227
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in Large Language Models (LLMs) have catalyzed the development of multi-agent systems (MAS) for complex reasoning tasks. However, existing MAS typically rely on pre-defined or pre-compiled communication topologies, which limits their flexibility and adaptability to dynamic task requirements. In this work, we propose Differentiable Mixture-of-Agents (DMoA), a self-evolving multi-agent framework that enables elastic and adaptive agent collaboration during inference. Instead of statically constructing workflows, DMoA dynamically routes and activates agents at each reasoning step, allowing the system to implicitly simulate diverse communication topologies and adapt to evolving demands. To achieve this, we design a differentiable, context-aware routing mechanism that leverages recurrent structures to incorporate historical and contextual information, producing sparse agent activations in a step-wise manner. Furthermore, we introduce predictive entropy as self-supervised signals to optimize the routing process, enabling efficient test-time adaptation without external annotations. Extensive experiments across 9 benchmarks demonstrate that DMoA achieves state-of-the-art performance while exhibiting strong efficiency, robustness, and ensembling capabilities.

Related papers

• DAIN: Dynamic Agent-Based Interaction Network for Efficient and Collaborative Multimodal Reasoning [21.622690972238615]
  We introduce the Dynamic Agent-based Interaction Network (DAIN), which reconceptualizes multimodal fusion as a dynamic, multi-agent collaborative process.<n>DAIN employs a context-aware Meta-Controller that schedules sparse activation of specialized interaction agents and orchestrates compressed inter-agent communication for consensus-building.
  arXiv  Detail & Related papers  (2026-06-29T12:04:01Z)
• AffectAgent: Collaborative Multi-Agent Reasoning for Retrieval-Augmented Multimodal Emotion Recognition [62.16431420189863]
  LLM-based multimodal emotion recognition relies on static parametric memory and often hallucinates when interpreting nuanced affective states.<n>We introduce AffectAgent, an affect-oriented multi-agent retrieval-augmented generation framework.<n>AffectAgent comprises three jointly optimized agents, namely a query planner, an evidence filter, and an emotion generator.
  arXiv  Detail & Related papers  (2026-04-14T13:49:19Z)
• Towards Adaptive, Scalable, and Robust Coordination of LLM Agents: A Dynamic Ad-Hoc Networking Perspective [31.81236449944822]
  RAPS is a reputation-aware publish-subscribe paradigm for adaptive, scalable, and robust coordination of LLM agents.<n>RAPS incorporates two coherent overlays: (i) Reactive Subscription, enabling agents to dynamically refine their intents; and (ii) Bayesian Reputation, empowering each agent with a local watchdog to detect and isolate malicious peers.
  arXiv  Detail & Related papers  (2026-02-08T15:26:02Z)
• ComAgent: Multi-LLM based Agentic AI Empowered Intelligent Wireless Networks [62.031889234230725]
  6G networks rely on complex cross-layer optimization.<n> manually translating high-level intents into mathematical formulations remains a bottleneck.<n>We present ComAgent, a multi-LLM agentic AI framework.
  arXiv  Detail & Related papers  (2026-01-27T13:43:59Z)
• Dynamic Generation of Multi-LLM Agents Communication Topologies with Graph Diffusion Models [99.85131798240808]
  We introduce a novel generative framework called textitGuided Topology Diffusion (GTD)<n>Inspired by conditional discrete graph diffusion models, GTD formulates topology synthesis as an iterative construction process.<n>At each step, the generation is steered by a lightweight proxy model that predicts multi-objective rewards.<n>Experiments show that GTD can generate highly task-adaptive, sparse, and efficient communication topologies.
  arXiv  Detail & Related papers  (2025-10-09T05:28:28Z)
• HiVA: Self-organized Hierarchical Variable Agent via Goal-driven Semantic-Topological Evolution [13.440964262446558]
  Hierarchical Variable Agent (HiVA) is a novel framework modeling agentic as self-organized graphs with the Semantic-Topological Evolution (STEV) algorithm.<n> Experiments on dialogue, coding, Longcontext Q&A, mathematical, and agentic benchmarks demonstrate improvements of 5-10% in task accuracy and enhanced resource efficiency.
  arXiv  Detail & Related papers  (2025-08-29T18:51:18Z)
• ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interaction [84.90394416593624]
  Agentic task-solving with Large Language Models (LLMs) requires multi-turn, multi-step interactions.<n>Existing simulation-based data generation methods rely heavily on costly autoregressive interactions between multiple agents.<n>We propose a novel Non-Autoregressive Iterative Generation framework, called ToolACE-MT, for constructing high-quality multi-turn agentic dialogues.
  arXiv  Detail & Related papers  (2025-08-18T07:38:23Z)
• Synchronization Dynamics of Heterogeneous, Collaborative Multi-Agent AI Systems [0.0]
  We present a novel interdisciplinary framework that bridges synchronization theory and multi-agent AI systems.<n>We adapt the Kuramoto model to describe the collective dynamics of heterogeneous AI agents engaged in complex task execution.
  arXiv  Detail & Related papers  (2025-08-17T10:16:41Z)
• AnyMAC: Cascading Flexible Multi-Agent Collaboration via Next-Agent Prediction [77.62279834617475]
  We propose a new framework that rethinks multi-agent coordination through a sequential structure rather than a graph structure.<n>Our method focuses on two key directions: (1) Next-Agent Prediction, which selects the most suitable agent role at each step, and (2) Next-Context Selection, which enables each agent to selectively access relevant information from any previous step.
  arXiv  Detail & Related papers  (2025-06-21T18:34:43Z)
• Multi-Agent Collaboration via Evolving Orchestration [55.574417128944226]
  Large language models (LLMs) have achieved remarkable results across diverse downstream tasks, but their monolithic nature restricts scalability and efficiency in complex problem-solving.<n>We propose a puppeteer-style paradigm for LLM-based multi-agent collaboration, where a centralized orchestrator ("puppeteer") dynamically directs agents ("puppets") in response to evolving task states.<n> Experiments on closed- and open-domain scenarios show that this method achieves superior performance with reduced computational costs.
  arXiv  Detail & Related papers  (2025-05-26T07:02:17Z)
• Neural Orchestration for Multi-Agent Systems: A Deep Learning Framework for Optimal Agent Selection in Multi-Domain Task Environments [0.4640835690336653]
  We propose MetaOrch, a neural orchestration framework for optimal agent selection in multi-domain task environments.<n>A novel fuzzy evaluation module scores agent responses along completeness, relevance, and confidence dimensions, generating soft supervision labels for training the orchestrator.<n>Experiments in simulated environments with heterogeneous agents demonstrate that our approach achieves 86.3% selection accuracy.
  arXiv  Detail & Related papers  (2025-05-03T02:58:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.