RouteRec: Strict Evaluation of Recommender-Agent Selection and Aggregation

• URL: http://arxiv.org/abs/2607.09908v1
• Date: Fri, 10 Jul 2026 18:58:52 GMT
• Title: RouteRec: Strict Evaluation of Recommender-Agent Selection and Aggregation
• Abstract summary: RouteRec is a framework that compares request-level hard selection with item-level learned aggregation.<n>On MovieLens-1M, the full quality oracle has substantial headroom (HR@10 = 0.584)<n>The resulting lesson is not that routing is solved, but item-level aggregation is the more promising action space.
• Score: 4.548136909830987
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recommender systems increasingly face a choice among heterogeneous agents -- collaborative filters, sequential models, content-based retrievers, and LLM-based rerankers -- yet no single agent is uniformly best. We study this choice as task-aware agent ranking under cost constraints using RouteRec, a framework that compares request-level hard selection with item-level learned aggregation over four traditional recommender agents and one LLM reranker agent. On MovieLens-1M, the full quality oracle has substantial headroom (HR@10 = 0.584), confirming that useful cross-agent signal exists. Under a leakage-free 5-fold out-of-fold protocol, however, hard selection remains below BM25 (0.223 vs. 0.254), and selective LLM escalation does not improve it. The same protocol yields a different outcome for learned aggregation: its cheap-only variant matches BM25 in HR and has a higher NDCG point estimate (0.123 vs. 0.114), while gated all-agent aggregation reaches HR@10 = 0.295 with 70.2\% LLM calls. The resulting lesson is not that routing is solved, but that request-level selection of one complete agent list is too coarse for this sparse fixed-candidate setting; item-level aggregation is the more promising action space.

Related papers

• Natural-Language-Guided Generator-Agnostic Shortlisting for Protein Binder Design [16.443448156662644]
  Modern de novo design generate many candidate protein binders, but wet-lab validation capacity remains limited.<n>We study whether LLMs can generate multi-metric ranking policies from precomputed structural-confidence and interface-quality proxy scores.
  arXiv  Detail & Related papers  (2026-08-21T05:32:26Z)
• SLMs as Multi-Agent Routers: A Progressive SFT and Reinforcement Learning Approach [35.31208793946578]
  Specialised retrieval agents typically surface higher quality results than general-purpose search.<n>Current approaches route queries based on inferred topic or intent, but it does not incorporate signal from retrieved content.<n>We address this by training a small language model via supervised fine-tuning followed by reinforcement learning.
  arXiv  Detail & Related papers  (2026-07-15T11:11:39Z)
• Multi-Agent Routing as Set-Valued Prediction: A WildChat Benchmark and Cost-Aware Evaluation [0.0]
  The benchmark introduced here contains 3,000 prompts over a fixed 12-agent catalog, with AI-assisted labels under a fixed schema.<n>The evaluation protocol combines set-level metrics (Precision, Recall, F1, Jaccard, and Exact Match), latency, an execution-oriented capability-coverage simulation, and a constrained weighted-routing setting.
  arXiv  Detail & Related papers  (2026-06-27T13:59:46Z)
• Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration [57.23877089929136]
  Graph-of-Agents (GoA) is a new graph-based framework for modeling multi-agent LLM communication.<n>GoA achieves superior performance using only 3 selected agents, outperforming recent multi-agent LLM baselines.
  arXiv  Detail & Related papers  (2026-04-18T21:13:03Z)
• Agent Q-Mix: Selecting the Right Action for LLM Multi-Agent Systems through Reinforcement Learning [68.85408801740228]
  We propose textbfAgent Q-Mix, a reinforcement learning framework that reformulates topology selection as a cooperative Multi-Agent Reinforcement Learning problem.<n>Our method learns decentralized communication decisions using Qmix value factorization, where each agent selects from a set of communication actions that jointly induce a round-wise communication graph.<n>Agent Q-Mix achieves the highest average accuracy compared to existing methods while demonstrating superior token efficiency and robustness against agent failure.
  arXiv  Detail & Related papers  (2026-04-01T00:38:24Z)
• When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines [0.0]
  Multi-agent LLM pipelines produce contradictory evidence on whether team diversity improves output quality.<n>We propose a resolution by identifying the selection bottleneck that determines whether diversity helps or hurts.<n>Our results suggest that selector quality may be a more impactful design lever than generator diversity in single-round generate-then-select pipelines.
  arXiv  Detail & Related papers  (2026-03-20T00:50:53Z)
• AgentSelect: Benchmark for Narrative Query-to-Agent Recommendation [39.61543921719145]
  AgentSelect is a benchmark that reframes agent selection as narrative query-to-agent recommendation.<n>It converts heterogeneous evaluation artifacts into unified, positive-only interaction data.<n>AgentSelect provides the first unified data and evaluation infrastructure for agent recommendation.
  arXiv  Detail & Related papers  (2026-03-04T06:17:51Z)
• Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools [0.17188280334580197]
  Reranking evaluations study how a reranker orders candidates returned by an upstream retriever.<n>This setup couples ranking behavior with retrieval quality, so differences in output cannot be attributed to the ranking policy alone.<n>We introduce a controlled diagnostic that isolates reranking by using Multi-News clusters as fixed evidence pools.
  arXiv  Detail & Related papers  (2026-02-20T21:07:32Z)
• RouteMoA: Dynamic Routing without Pre-Inference Boosts Efficient Mixture-of-Agents [91.0187958746262]
  RouteMoA is an efficient mixture-of-agents framework with dynamic routing.<n>It employs a lightweight scorer to perform initial screening by predicting coarse-grained performance from the query.<n>It refines these scores through lightweight self- and cross-assessment based on existing model outputs, providing posterior correction without additional inference.
  arXiv  Detail & Related papers  (2026-01-26T04:22:22Z)
• OptAgent: Optimizing Query Rewriting for E-commerce via Multi-Agent Simulation [1.3722079106827219]
  OptAgent is a novel framework that combines multi-agent simulations with genetic algorithms to verify and optimize queries for e-commerce queries.<n>We evaluate OptAgent on a dataset of 1000 real-world e-commerce queries in five different categories.
  arXiv  Detail & Related papers  (2025-10-04T10:41:09Z)
• Collab: Controlled Decoding using Mixture of Agents for LLM Alignment [90.6117569025754]
  Reinforcement learning from human feedback has emerged as an effective technique to align Large Language models.<n>Controlled Decoding provides a mechanism for aligning a model at inference time without retraining.<n>We propose a mixture of agent-based decoding strategies leveraging the existing off-the-shelf aligned LLM policies.
  arXiv  Detail & Related papers  (2025-03-27T17:34:25Z)
• MALT: Improving Reasoning with Multi-Agent LLM Training [67.76186488361685]
  MALT (Multi-Agent LLM Training) is a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps.<n>On MATH, GSM8K, and CSQA, MALT surpasses the same baseline LLM with a relative improvement of 15.66%, 7.42%, and 9.40% respectively.
  arXiv  Detail & Related papers  (2024-12-02T19:30:36Z)
• Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting [65.00288634420812]
  Pairwise Ranking Prompting (PRP) is a technique to significantly reduce the burden on Large Language Models (LLMs) Our results are the first in the literature to achieve state-of-the-art ranking performance on standard benchmarks using moderate-sized open-sourced LLMs.
  arXiv  Detail & Related papers  (2023-06-30T11:32:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.