Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains

• URL: http://arxiv.org/abs/2607.18438v1
• Date: Mon, 20 Jul 2026 18:46:17 GMT
• Title: Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains
• Abstract summary: Relay-Bench is an unsaturated, holistic, text-only benchmark.<n>It measures LLMs' ability to complete an assortment of tasks from distinct domains in a single prompt.<n>The leading model, GPT-5.5 (xHigh), scores 43.3%.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Introducing Relay-Bench, an unsaturated, holistic, text-only benchmark that measures LLMs' ability to complete an assortment of tasks from distinct domains in a single prompt. The leading model, GPT-5.5 (xHigh), scores 43.3%. The test set entirely consists of composite problems: groups of single-domain subproblems that are strung together into challenges that require reasoning across multiple domains in combination. Many of these problems then have layers of complexity added through prompt encoding and deliberate context bloat. Domains tested include visual reasoning, coding, math, information extraction (with a focus on web search), problem-solving, general knowledge, and data analysis. No restrictions are imposed outside of the model harness, and models are explicitly encouraged to leverage code-execution, web searches, and all available tools. All problems are composed of two to thirteen subproblems and do not require multi-modal input or output.

Related papers

• M$^3$-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering [16.14696376349009]
  M$3$-VQA is a knowledge-based Visual Question Answering (VQA) benchmark.<n>It introduces diverse multi-entity questions involving multiple distinct entities from both visual and textual sources.<n>It requires models to perform both sequential and parallel multi-hop reasoning across multiple documents.
  arXiv  Detail & Related papers  (2026-04-28T01:57:22Z)
• Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation [56.69064935192318]
  Multi-hop Question Answering (MHQA) adds layers of complexity to question answering, making it more challenging.<n>This paper explores how Language Models respond to multi-hop questions by permuting search results (retrieved documents) under various configurations.
  arXiv  Detail & Related papers  (2025-05-16T23:29:47Z)
• Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent [92.5712549836791]
  Multimodal Retrieval Augmented Generation (mRAG) plays an important role in mitigating the "hallucination" issue inherent in multimodal large language models (MLLMs)<n>We propose the first self-adaptive planning agent for multimodal retrieval, OmniSearch.
  arXiv  Detail & Related papers  (2024-11-05T09:27:21Z)
• MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains [54.117238759317004]
  Massive Multitask Agent Understanding (MMAU) benchmark features comprehensive offline tasks that eliminate the need for complex environment setups. It evaluates models across five domains, including Tool-use, Directed Acyclic Graph (DAG) QA, Data Science and Machine Learning coding, Contest-level programming and Mathematics. With a total of 20 meticulously designed tasks encompassing over 3K distinct prompts, MMAU provides a comprehensive framework for evaluating the strengths and limitations of LLM agents.
  arXiv  Detail & Related papers  (2024-07-18T00:58:41Z)
• Navigating the Labyrinth: Evaluating LLMs' Ability to Reason About Search Problems [62.76627483915117]
  Large Language Models (LLMs) have recently achieved impressive performance in math and reasoning benchmarks.<n>We introduce a new benchmark, SearchBench, which contains 11 unique search problems inspired by intuitive puzzles.<n>We show that using step-by-step, language-only reasoning, even the most advanced LLMs fail to solve SearchBench.
  arXiv  Detail & Related papers  (2024-06-18T00:44:58Z)
• Evaluating LLMs with Multiple Problems at once [9.173325772800341]
  This paper shows the benefits and fruitfulness of evaluating LLMs with multiple problems at once.<n>We introduce a new benchmark called ZeMPE (Zero-shot Multi-Problem Evaluation), comprising 53,100 zero-shot multi-problem prompts.<n>Our results show that LLMs are capable of handling multiple problems from a single data source as well as handling them separately, but there are conditions this multiple problem handling capability falls short.
  arXiv  Detail & Related papers  (2024-06-16T02:52:32Z)
• FCoReBench: Can Large Language Models Solve Challenging First-Order Combinatorial Reasoning Problems? [25.352721856952655]
  First-order reasoning problems can be instantiated with infinite number of problem instances of varying sizes.<n>We present FCoReBench, a dataset of 40 such challenging problems, along with scripts to generate problem instances of varying sizes and automatically verify and generate their solutions.<n>We propose SymPro-LM, which combines LLMs with both symbolic solvers and program interpreters, along with feedback from a few solved examples, to achieve huge performance gains.
  arXiv  Detail & Related papers  (2024-02-04T20:56:09Z)
• AGIBench: A Multi-granularity, Multimodal, Human-referenced, Auto-scoring Benchmark for Large Language Models [3.518832148294879]
  How to evaluate the question-solving abilities of large language models like ChatGPT is a hot-spot but challenging issue. We propose AGIBench -- a multi-granularity, multimodal, human-referenced, and auto-scoring benchmarking methodology for LLMs.
  arXiv  Detail & Related papers  (2023-09-05T13:43:37Z)
• Faith and Fate: Limits of Transformers on Compositionality [109.79516190693415]
  We investigate the limits of transformer large language models across three representative compositional tasks. These tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer. Our empirical findings suggest that transformer LLMs solve compositional tasks by reducing multi-step compositional reasoning into linearized subgraph matching.
  arXiv  Detail & Related papers  (2023-05-29T23:24:14Z)
• Successive Prompting for Decomposing Complex Questions [50.00659445976735]
  Recent works leverage the capabilities of large language models (LMs) to perform complex question answering in a few-shot setting. We introduce Successive Prompting'', where we iteratively break down a complex task into a simple task, solve it, and then repeat the process until we get the final solution. Our best model (with successive prompting) achieves an improvement of 5% absolute F1 on a few-shot version of the DROP dataset.
  arXiv  Detail & Related papers  (2022-12-08T06:03:38Z)
• OLGA : An Ontology and LSTM-based approach for generating Arithmetic Word Problems (AWPs) of transfer type [0.0]
  Machine generation of Arithmetic Word Problems (AWPs) is challenging as they express quantities and mathematical relationships. We propose a system, OLGA, to generate consistent word problems of TC (Transfer-Case) type. We use an extension of TC-Ontology, proposed by us previously, to determine the consistency of problems.
  arXiv  Detail & Related papers  (2022-11-22T10:42:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.