Rubric-Guided Process Reward for Stepwise Model Routing

• URL: http://arxiv.org/abs/2605.29310v1
• Date: Thu, 28 May 2026 03:42:24 GMT
• Title: Rubric-Guided Process Reward for Stepwise Model Routing
• Abstract summary: Stepwise model routing improves the efficiency of Large Reasoning Models (LRMs) by assigning each reasoning step to a suitable model.<n>Recent methods formulate routing as a sequential decision process and train the router with reinforcement learning.<n>We propose RoRo, a rubric-guided process reward framework for stepwise model routing.
• Score: 18.58577344135632
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Stepwise model routing improves the efficiency of Large Reasoning Models (LRMs) by assigning each reasoning step to a suitable model. Recent methods formulate routing as a sequential decision process and train the router with reinforcement learning. However, although they model routing as a process, they still supervise the router with outcome rewards. Such rewards only reflect final answer correctness and fail to evaluate intermediate routing decisions, which can weaken performance and generalization. To address this gap, we propose RoRo, a rubric-guided process reward framework for stepwise model routing. RoRo first collects diverse routing trajectories and constructs preference pairs based on outcome, cost, and process quality. It then trains a Rubricor to generate a query-specific evaluation rubric and a Judge to score routing trajectories under this rubric through alternating optimization. The resulting process rewards are combined with outcome rewards to optimize the routing policy via GRPO. Experiments on five reasoning benchmarks under both same-family and cross-family settings show that RoRo consistently outperforms strong baselines and achieves better accuracy and cost trade-offs.

Related papers

• SCOPE-Router: Cost-Aware Open-Set VLM Routing for Execution-Oriented Tasks [26.364477396804194]
  Model routing aims to select the most suitable model from a candidate pool for each query, balancing quality and cost.<n>Existing VLM routing research is limited to traditional VQA evaluation, lacks systematic calibration optimization for open-set scenarios.<n>We address these limitations with three contributions: (1)VLM-ExecBench, the first execution-oriented VLM routing benchmark covering Code, Agentic, and Search domains; (2) SCOPE, a dual-tower router that matches queries to model behavior profiles constructed via hybrid calibration; and (3) CRM+RCCR, an architecture-agnostic cost-aware objective that encodes cost preference into continuous relevance
  arXiv  Detail & Related papers  (2026-08-12T14:49:28Z)
• VDAR-Router: Adaptive LLMs Routing via Verbalized Query Difficulty Analysis Retrieval [16.067071673823417]
  We propose VDAR-the, a difficulty-aware retrieval-based routing framework.<n>For each input query, VDAR-the first generates an explicit difficulty analysis.<n>It then retrieves historical examples with similar difficulty profiles.
  arXiv  Detail & Related papers  (2026-07-20T16:00:45Z)
• Agent-as-a-Router: Agentic Model Routing for Coding Tasks [54.00153674547733]
  Agent-as-a-router (AC) formalizes routing as a C-A-F loop (Context->Action->FeedbackContext)<n>AC achieves the lowest cumulative regret on in-distribution tasks and generalizes to out-of-distribution agentic-programming tasks.
  arXiv  Detail & Related papers  (2026-06-22T06:37:31Z)
• RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing [41.435534676485936]
  RouteJudge is an online pairwise preference evaluation framework for LLM routing systems.<n>Multiple routing strategies independently recommend model models under the same model pool and budget constraints.<n>The resulting user preferences are attributed back to the routing strategies behind the compared responses.
  arXiv  Detail & Related papers  (2026-06-17T07:35:10Z)
• The Routing Plateau: Understanding and Breaking the Accuracy Limits of LLM Routers [19.847272668359707]
  LLM routing is a popular approach to improve the cost-quality trade-off of LLM services by dynamically selecting a model for each query.<n>Recent work has explored a broad range of routing methods, including clustering-based routers, learned classifiers, pairwise ranking, and confidence-based approaches.<n>Our study reveals a consistent phenomenon that we call the routing plateau: many methods, including kNN, achieve very similar accuracy and converge to a narrow performance range that remains far below the oracle router.
  arXiv  Detail & Related papers  (2026-05-27T19:29:01Z)
• Beyond Query Memorization: Large Language Model Routing with Query Decomposition and Historical Matching [6.980897284458787]
  DecoR recasts the routing task as a matching process of sifting similar queries from historical logs.<n>To enhance matching accuracy, we introduce a query capability that decouples linguistic surface forms from task-intrinsic requirements.
  arXiv  Detail & Related papers  (2026-05-25T08:12:58Z)
• Route Before Retrieve: Activating Latent Routing Abilities of LLMs for RAG vs. Long-Context Selection [57.3886742625188]
  Pre-Route is a proactive routing framework that performs structured reasoning before answering.<n>Our study shows three key findings: (i) LLMs possess latent routing ability that can be reliably elicited with guidelines; (ii) linear probes reveal that structured prompts sharpen the separability of the "optimal routing" in representation space; and (iii) distillation transfers this reasoning structure to smaller models for lightweight deployment.
  arXiv  Detail & Related papers  (2026-05-11T09:10:55Z)
• Efficient and Interpretable Multi-Agent LLM Routing via Ant Colony Optimization [58.59491516762626]
  We propose AMRO-S, an efficient and interpretable routing framework for Multi-Agent Systems (MAS)<n>AMRO-S models MAS routing as a semantic-conditioned path selection problem, enhancing routing performance through three key mechanisms.<n>Extensive experiments on five public benchmarks and high-concurrency stress tests demonstrate that AMRO-S consistently improves the quality--cost trade-off over strong routing baselines.
  arXiv  Detail & Related papers  (2026-03-13T12:26:05Z)
• Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offs [69.2486294522259]
  BaRP is a Bandit Routing-feedback with Preferences approach that trains under the same partial-feedback restriction as deployment.<n> Framed as a contextual bandit over prompt features and a user preference vector, our method simulates an online feedback setting during training and adapts its routing decisions to each new prompt.
  arXiv  Detail & Related papers  (2025-10-08T18:24:59Z)
• Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning [27.70756702796812]
  We present textbf generalization-R1, a reinforcement learning framework that formulates multi-LLM routing and aggregation as a sequential decision process.<n>To facilitate learning, we employ a lightweight rule-based reward comprising format rewards, final outcome rewards, and a novel cost reward for optimizing the balance between performance and cost.
  arXiv  Detail & Related papers  (2025-06-10T17:56:45Z)
• Learning to Route Queries Across Knowledge Bases for Step-wise Retrieval-Augmented Reasoning [60.84901522792042]
  Multimodal Retrieval-Augmented Generation (MRAG) has shown promise in mitigating hallucinations in Multimodal Large Language Models (MLLMs)<n>We propose R1, a novel MRAG framework that learns to decide when and where to retrieve knowledge based on the evolving reasoning state.<n>R1- can adaptively and effectively leverage diverse KBs, reducing unnecessary retrievals and improving both efficiency and accuracy.
  arXiv  Detail & Related papers  (2025-05-28T08:17:57Z)
• CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing [74.14816777318033]
  Collaborative Inference with Token-lEvel Routing (CITER) is a framework that enables efficient collaboration between small and large language models.<n>We formulate router training as a policy optimization, where the router receives rewards based on both the quality of predictions and the inference costs of generation.<n>Our experiments show that CITER reduces the inference costs while preserving high-quality generation, offering a promising solution for real-time and resource-constrained applications.
  arXiv  Detail & Related papers  (2025-02-04T03:36:44Z)
• Inverse Optimization for Routing Problems [3.282021317933024]
  We propose a method for learning decision-makers' behavior in routing problems using Inverse Optimization (IO) Our examples and results showcase the flexibility and real-world potential of the proposed IO methodology to learn from decision-makers' decisions in routing problems.
  arXiv  Detail & Related papers  (2023-07-14T14:03:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.