SciOrch: Learning to Orchestrate Expert LLMs for Solving Frontier Multimodal Scientific Reasoning Tasks

• URL: http://arxiv.org/abs/2606.15872v1
• Date: Sun, 14 Jun 2026 15:45:34 GMT
• Title: SciOrch: Learning to Orchestrate Expert LLMs for Solving Frontier Multimodal Scientific Reasoning Tasks
• Abstract summary: We present SciOrch, a framework that trains a lightweight 8B model to orchestrate frontier LLMs for scientific reasoning.<n>On a 240-question test set spanning SGI-Reasoning and Scientists' First exam, SciOrch reaches 56.66% average accuracy.<n>It also attains the best accuracy on both SGI and SFE with less than half the API cost of typical multi-agent methods.
• Score: 65.14594927081983
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Frontier scientific reasoning remains a major challenge for large language models (LLMs), where even the strongest commercial systems fall short of expert-level performance. A closer look at model behavior reveals substantial complementarity that single-model evaluation hides: different frontier models excel on different question types, and no single model captures the full picture. We present SciOrch, a framework that trains a lightweight 8B model to orchestrate frontier LLMs for scientific reasoning. The orchestrator decomposes each question, delegates sub-problems to selected commercial models through API calls, and synthesizes a final answer. Training such an orchestrator is fundamentally harder than conventional agentic RL: each action triggers an API call that is expensive in both dollar cost and latency, making standard online rollouts infeasible. We address this with MCTS-based approach, producing diverse orchestration trajectories, extracting per-node single-turn samples, and optimizing the orchestrator with GRPO-style training. On a 240-question test set spanning SGI-Reasoning and Scientists' First Exam, SciOrch reaches 56.66% average accuracy, outperforming the strongest single commercial model by 3.74% and the strongest multi-agent baseline by 3.33%. It also attains the best accuracy on both SGI and SFE with less than half the API cost of typical multi-agent methods.

Related papers

• Reward Modeling for Multi-Agent Orchestration [43.79233309038303]
  Multi-Agent Systems (MAS) built on Large Language Models (LLMs) require effective orchestration to coordinate specialized agents.<n>We propose Orchestration Reward Modeling (OrchRM), a self-supervised framework for evaluating orchestration quality without human annotations.
  arXiv  Detail & Related papers  (2026-06-11T17:16:24Z)
• LLM-driven design of physics-constrained constitutive models: two agents are better than one [0.0]
  We introduce the first multi-agent LLM-driven approach for model generation.<n>A Creator agent proposes a model tailored to the data, while an Inspector agent audits each proposal against nine physical constraints.<n>We demonstrate this concept with artificial neural networks (CANNs) and benchmark it on brain tissue, experimental rubber, and synthetic rubber.
  arXiv  Detail & Related papers  (2026-05-22T15:27:09Z)
• CascadeDebate: Multi-Agent Deliberation for Cost-Aware LLM Cascades [3.2562960801091094]
  CascadeDebate is a multi-agent system that coordinates models with human experts to balance accuracy, cost, and abstention under uncertainty.<n>Our architecture alternates single-model inference with selective multi-agent deliberation across model scales, culminating in human experts as the final fallback.<n>Across five benchmarks spanning science, medicine, and general knowledge, CascadeDebate outperforms strong single-model cascades and standalone multi-agent systems by up to 26.75 percent.
  arXiv  Detail & Related papers  (2026-04-14T04:26:39Z)
• RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty [102.02839046225468]
  RankLLM is a novel framework designed to quantify both question difficulty and model competency.<n>We evaluate 30 models on 35,550 questions across multiple domains.
  arXiv  Detail & Related papers  (2026-02-12T21:28:46Z)
• CARGO: A Framework for Confidence-Aware Routing of Large Language Models [6.002503434201551]
  We introduce CARGO, a lightweight, confidence-aware framework for dynamic large language models (LLMs) selection.<n>CARGO employs a single embedding-based regressor trained on LLM-judged pairwise comparisons to predict model performance.<n>CARGO achieves a top-1 routing accuracy of 76.4% and win rates ranging from 72% to 89% against individual experts.
  arXiv  Detail & Related papers  (2025-09-18T12:21:30Z)
• Multi-Model Synthetic Training for Mission-Critical Small Language Models [0.0]
  We present a novel approach that achieves a 261x cost reduction for maritime intelligence.<n>Our method transforms 3.2 billion Automatic Identification System (AIS) vessel tracking records into 21,543 synthetic question and answer pairs.<n>The resulting fine-tuned Qwen2.5-7B model achieves 75% accuracy on maritime tasks, while being substantially cheaper than using a larger model for inference.
  arXiv  Detail & Related papers  (2025-09-16T13:04:48Z)
• CTTS: Collective Test-Time Scaling [58.564620942591866]
  Test-time scaling (TTS) has emerged as a promising, training-free approach for enhancing large language model (LLM) performance.<n>We introduce Collective Test-Time Scaling (CTTS) to overcome the dominant single test-time scaling (STTS) paradigm.<n>CTTS-MM is a novel framework that operationalizes multi-agent and multi-reward collaboration.
  arXiv  Detail & Related papers  (2025-08-05T11:19:08Z)
• APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay [86.01901238059261]
  APIGen-MT is a framework that generates verifiable and diverse multi-turn agent data.<n>We train a family of models -- the xLAM-2-fc-r series with sizes ranging from 1B to 70B parameters.<n>Our models outperform frontier models such as GPT-4o and Claude 3.5 on $tau$-bench and BFCL benchmarks.
  arXiv  Detail & Related papers  (2025-04-04T17:13:57Z)
• From Dense to Dynamic: Token-Difficulty Driven MoEfication of Pre-Trained LLMs [37.50902921493273]
  Training large language models (LLMs) for different inference constraints is computationally expensive.<n> DynaMoE adapts a pre-trained dense LLM to a token-difficulty-driven Mixture-of-Experts model with minimal fine-tuning cost.<n>Our method achieves similar aggregated accuracy across downstream tasks, despite using only $frac19textth$ of their fine-tuning cost.
  arXiv  Detail & Related papers  (2025-02-17T21:12:57Z)
• Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training [73.90260246781435]
  We present Lory, the first approach that scales such architectures to autoregressive language model pre-training. We show significant performance gains over parameter-matched dense models on both perplexity and a variety of downstream tasks. Despite segment-level routing, Lory models achieve competitive performance compared to state-of-the-art MoE models with token-level routing.
  arXiv  Detail & Related papers  (2024-05-06T03:06:33Z)
• Toward Inference-optimal Mixture-of-Expert Large Language Models [55.96674056805708]
  We study the scaling law of MoE-based large language models (LLMs) We find that MoEs with a few (4/8) experts are the most serving efficient solution under the same performance, but costs 2.5-3.5x more in training. We propose to amend the scaling law of MoE by introducing inference efficiency as another metric besides the validation loss.
  arXiv  Detail & Related papers  (2024-04-03T16:33:42Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.