When (and How) to Trust the Expert: Diagnosing Query-Time Expert-Guided Reinforcement Learning

• URL: http://arxiv.org/abs/2605.09109v1
• Date: Sat, 09 May 2026 18:48:17 GMT
• Title: When (and How) to Trust the Expert: Diagnosing Query-Time Expert-Guided Reinforcement Learning
• Abstract summary: Continuous-control problems ship with a competent but suboptimal controller (a tuned PID, a hand-designed gait)<n>A growing family of methods uses such controllers as queryable experts during RL, but each method has been proposed in isolation, on a different benchmark, without imperfect-expert testing.<n>We harmonize the comparison on a shared SAC backbone, common HPO and evaluation protocols, 100/50 seeds per (env, method), and a degradation sweep over expert undertuning, action bias, and observation noise.
• Score: 4.913013713982676
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many continuous-control problems ship with a competent but suboptimal controller (a tuned PID, a hand-designed gait). A growing family of methods uses such controllers as queryable experts during RL, but each method has been proposed in isolation, on a different benchmark, without imperfect-expert testing. We harmonize the comparison on a shared SAC backbone, common HPO and evaluation protocols, 100/50 seeds per (env, method), and a degradation sweep over expert undertuning, action bias, and observation noise. The comparison surfaces three failure modes single-paper evaluations miss: (F1) a critic blind spot under argmax-plus-bootstrap that drags IBRL below no-expert SAC on experts close to the no-expert-RL ceiling (RL-near-ceiling, distinct from the absolute physical ceiling); (F2) residual saturation on far-from-optimal experts; and (F3) warm-start buffer poisoning that collapses training-time-handoff methods under deployment-time expert undertuning. No single method dominates: each wins on one task-structure regime and fails predictably elsewhere; on RL-near-ceiling experts (FourTank, GlassFurnace) no query-time method clears the expert within our 1M-step budget, leaving open whether this is a fundamental wall or a budget effect. We convert the spread into a testable decision rule keyed on three pre-training observables (expert quality, task termination, perturbation type). The benchmark, taxonomy, and decision rule are the primary contribution; we additionally describe EDGE, a softmax-over-ensemble-LCB design point used to demonstrate that both axes the taxonomy points to (gate form, scoring rule) are individually exploitable.

Related papers

• RAPTOR: Role-Aware Private Training for Mixture-of-Experts [68.5626338219096]
  RAPTOR is a Role-Aware Private Training framework.<n>RAPTOR alternates shared and expert optimization and targets each failure directly.<n> Experiments on Switch Transformer and OLMoE fine-tuning across GLUE tasks, and on DeepSeek-VL2-Tiny, show consistent gains over standard DP baselines.
  arXiv  Detail & Related papers  (2026-09-04T23:16:12Z)
• Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation [64.51088225051959]
  Heterogeneous AI systems can improve quality and efficiency by routing queries to the specialist who can answer most effectively at the lowest cost.<n>We formalize this tradeoff as an instance of Pandora's Box, the classical problem of optimal search with costly inspection.<n>Under a Gaussian signal model, the resulting policies have closed-form value-of-information expressions that determine, for each specialist and input, whether refining the value estimate is worth its cost.
  arXiv  Detail & Related papers  (2026-08-20T17:54:37Z)
• PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception [75.70273182182397]
  PerceptionRubrics is a rubric-based evaluation framework.<n>It addresses the gap between saturated benchmark scores and real-world brittleness.<n>It pairs 1,038 information-dense images with over 10,000 instance-specific rubrics.
  arXiv  Detail & Related papers  (2026-06-26T17:59:15Z)
• Split the Differences, Pool the Rest: Provably Efficient Multi-Objective Imitation [49.86232017439639]
  We introduce Multi-Output Augmented Behavioral Cloning (MA-BC)<n>MA-BC partitions divergent expert data while pooling state-action pairs where no behavior conflict is observed.<n>We establish a novel lower bound for multi-objective imitation learning, demonstrating that MA-BC is minimax optimal.
  arXiv  Detail & Related papers  (2026-05-12T11:49:08Z)
• Beyond Augmented-Action Surrogates for Multi-Expert Learning-to-Defer [6.792743621449621]
  Multi-expert learning-to-defer surrogates are statistically consistent, yet they can underfit, suppress useful experts, or degrade as the expert pool grows.<n>We trace these failures to a shared architectural choice: casting classes and experts as actions inside one augmented prediction geometry.<n>We introduce a decoupled surrogate that estimates the class posterior with a softmax and each expert utility with an independent sigmoid.
  arXiv  Detail & Related papers  (2026-04-10T15:27:23Z)
• Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedback [2.4352490146713364]
  We introduce TriTrust-PBRL, a unified framework that jointly learns a shared reward model and expert-specific trust parameters from multi-expert preference feedback.<n>TTP achieves state-of-the-art robustness, maintaining near-oracle performance under adversarial corruption while standard PBRL methods fail catastrophically.
  arXiv  Detail & Related papers  (2026-01-26T18:21:48Z)
• Sample-Efficient Expert Query Control in Active Imitation Learning via Conformal Prediction [2.344992278528697]
  We present Conformalized Rejection Sampling for Active Imitation Learning (CRSAIL)<n>CRSAIL scores state novelty by the distance to the $K$-th nearest expert state.<n>It reduces total expert queries by up to 96% vs. DAgger and up to 65% vs. prior AIL methods.
  arXiv  Detail & Related papers  (2025-11-29T11:58:21Z)
• Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations [48.890534958441016]
  This study investigates domain specialization and expert redundancy in large-scale MoE models.<n>We propose a simple yet effective pruning framework, EASY-EP, to identify and retain only the most relevant experts.<n>Experiments on DeepSeek-R1 and DeepSeek-V3-0324 show that our method can achieve comparable performances and $2.99times$ throughput under the same memory budget with full model with only half the experts.
  arXiv  Detail & Related papers  (2025-04-09T11:34:06Z)
• Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations [86.90549830760513]
  Sparsely activated Mixture-of-Experts (SMoE) has shown promise in scaling up the learning capacity of neural networks.<n>We propose MoE Experts Compression Suite (MC-Suite) to provide a benchmark for estimating expert importance from diverse perspectives.<n>We present an experimentally validated conjecture that, during expert dropping, SMoEs' instruction-following capabilities are predominantly hurt.
  arXiv  Detail & Related papers  (2025-04-08T00:49:08Z)
• Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles [0.966840768820136]
  We study the statistical properties of learning to defer (L2D) to multiple experts. We address the open problems of deriving a consistent surrogate loss, confidence calibration, and principled ensembling of experts.
  arXiv  Detail & Related papers  (2022-10-30T21:27:29Z)
• Blind Exploration and Exploitation of Stochastic Experts [7.106986689736826]
  We present blind exploration and exploitation (BEE) algorithms for identifying the most reliable expert based on formulations that employ posterior sampling, upper-confidence bounds, empirical Kullback-Leibler divergence, and minmax methods for the multi-armed bandit problem. We propose an empirically realizable measure of expert competence that can be instantaneously using only the opinions of other experts.
  arXiv  Detail & Related papers  (2021-04-02T15:02:02Z)
• Confidence-Budget Matching for Sequential Budgeted Learning [69.77435313099366]
  We formalize decision-making problems with querying budget. We consider multi-armed bandits, linear bandits, and reinforcement learning problems. We show that CBM based algorithms perform well in the presence of adversity.
  arXiv  Detail & Related papers  (2021-02-05T19:56:31Z)
• Prediction with Corrupted Expert Advice [67.67399390910381]
  We prove that a variant of the classical Multiplicative Weights algorithm with decreasing step sizes achieves constant regret in a benign environment. Our results reveal a surprising disparity between the often comparable Follow the Regularized Leader (FTRL) and Online Mirror Descent (OMD) frameworks.
  arXiv  Detail & Related papers  (2020-02-24T14:39:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.