LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning

• URL: http://arxiv.org/abs/2604.14140v1
• Date: Wed, 15 Apr 2026 17:58:05 GMT
• Title: LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning
• Abstract summary: LongCoT is a scalable benchmark of 2,500 expert-designed problems spanning chemistry, mathematics, computer science, chess, and logic.<n>LongCoT provides a rigorous measure of long-horizon reasoning, tracking the ability of frontier models to reason reliably over extended periods.
• Score: 50.27907326876949
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As language models are increasingly deployed for complex autonomous tasks, their ability to reason accurately over longer horizons becomes critical. An essential component of this ability is planning and managing a long, complex chain-of-thought (CoT). We introduce LongCoT, a scalable benchmark of 2,500 expert-designed problems spanning chemistry, mathematics, computer science, chess, and logic to isolate and directly measure the long-horizon CoT reasoning capabilities of frontier models. Problems consist of a short input with a verifiable answer; solving them requires navigating a graph of interdependent steps that span tens to hundreds of thousands of reasoning tokens. Each local step is individually tractable for frontier models, so failures reflect long-horizon reasoning limitations. At release, the best models achieve <10% accuracy (GPT 5.2: 9.8%; Gemini 3 Pro: 6.1%) on LongCoT, revealing a substantial gap in current capabilities. Overall, LongCoT provides a rigorous measure of long-horizon reasoning, tracking the ability of frontier models to reason reliably over extended periods.

Related papers

• LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark [24.104346815675886]
  LongBench Pro is a more realistic benchmark of 1,500 naturally occurring long-context samples in English and Chinese.<n>It supports fine-grained analysis with task-specific metrics and a multi-dimensional taxonomy of context requirement.<n>LongBench Pro provides a robust testbed for advancing long-context understanding.
  arXiv  Detail & Related papers  (2026-01-06T10:01:59Z)
• DTS: Enhancing Large Reasoning Models via Decoding Tree Sketching [54.98126916293868]
  Large Reasoning Models (LRMs) produce excessively long chain-of-thought traces that degrade accuracy.<n>We propose a model-agnostic decoding framework that sketches the reasoning space by branching at high-entropy tokens and applies early stopping to select the shortest completed reasoning path.<n>This approach approximates the optimal solution that enhances both efficiency and accuracy, without requiring additional training or supervision.
  arXiv  Detail & Related papers  (2025-11-01T17:41:28Z)
• LongReasonArena: A Long Reasoning Benchmark for Large Language Models [94.58086553970101]
  LongReasonArena is a benchmark designed to assess the long reasoning capabilities of Large Language Models (LLMs)<n>By controlling the inputs, the required reasoning length can be arbitrarily scaled, reaching up to 1 million tokens of reasoning for the most challenging tasks.
  arXiv  Detail & Related papers  (2025-08-26T18:41:53Z)
• Adaptive Deep Reasoning: Triggering Deep Thinking When Needed [28.575411507835973]
  Large language models (LLMs) have shown impressive capabilities in handling complex tasks through long-chain reasoning.<n>We propose a novel approach that autonomously switches between short and long-chain reasoning chains based on problem complexity.<n>This advancement enhances the practicality of reasoning in large language models for real-world applications.
  arXiv  Detail & Related papers  (2025-05-26T15:08:51Z)
• SCoRE: Benchmarking Long-Chain Reasoning in Commonsense Scenarios [33.72114830484246]
  We introduce SCoRE (Scenario-based Commonsense Reasoning Evaluation), a benchmark that synthesizes multi-hop questions from scenario schemas of entities, relations, and logical rules.<n>SCoRE contains 100k bilingual (Chinese-English) multiple-choice questions whose reasoning chains span 2-11 hops and are grouped into various difficulty levels.
  arXiv  Detail & Related papers  (2025-03-08T13:40:10Z)
• Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning? [57.17826305464394]
  o1-like models produce the long Chain-of-Thought (CoT) reasoning steps to improve the reasoning abilities of existing Large Language Models (LLMs)<n>We introduce the DeltaBench, including the generated long CoTs from different o1-like models for different reasoning tasks.<n>Based on DeltaBench, we first perform fine-grained analysis of the generated long CoTs to discover the effectiveness and efficiency of different o1-like models.
  arXiv  Detail & Related papers  (2025-02-26T17:59:27Z)
• BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillation [88.77999917897702]
  o1 from OpenAI has demonstrated remarkable reasoning capabilities.<n>Many teams have attempted to replicate its LongCoT and reasoning capabilities.<n>This paper introduces a novel approach to enable LLM's LongCoT capacity without distillation from o1-like models or expensive human annotations.
  arXiv  Detail & Related papers  (2025-02-06T08:19:59Z)
• LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks [74.96182906307654]
  This paper introduces LongBench v2, a benchmark designed to assess the ability of LLMs to handle long-context problems.<n>LongBench v2 consists of 503 challenging multiple-choice questions, with contexts ranging from 8k to 2M words, across six major task categories.<n>We employ both automated and manual review processes to maintain high quality and difficulty, resulting in human experts achieving only 53.7% accuracy under a 15-minute time constraint.
  arXiv  Detail & Related papers  (2024-12-19T18:59:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.