ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Models

• URL: http://arxiv.org/abs/2603.19515v1
• Date: Thu, 19 Mar 2026 22:45:18 GMT
• Title: ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Models
• Abstract summary: Large language models (LLMs) with advanced cognitive capabilities are emerging as agents for various reasoning and planning tasks.<n>Recent studies have explored travel planning as a medium to integrate various verbal reasoning tasks into real-world contexts.<n>We introduce ItinBench, a benchmark that features one task of spatial reasoning, i.e., route optimization, into trip itinerary planning.
• Score: 8.453056552142447
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) with advanced cognitive capabilities are emerging as agents for various reasoning and planning tasks. Traditional evaluations often focus on specific reasoning or planning questions within controlled environments. Recent studies have explored travel planning as a medium to integrate various verbal reasoning tasks into real-world contexts. However, reasoning tasks extend beyond verbal reasoning alone, and a comprehensive evaluation of LLMs requires a testbed that incorporates tasks from multiple cognitive domains. To address this gap, we introduce ItinBench, a benchmark that features one task of spatial reasoning, i.e., route optimization, into trip itinerary planning while keeping the traditional verbal reasoning tasks. ItinBench evaluates various LLMs across diverse tasks simultaneously, including Llama 3.1 8B, Mistral Large, Gemini 1.5 Pro, and GPT family. Our findings reveal that LLMs struggle to maintain high and consistent performance when concurrently handling multiple cognitive dimensions. By incorporating tasks from distinct human-level cognitive domains, ItinBench provides new insights into building more comprehensive reasoning testbeds that better reflect real-world challenges. The code and dataset: https://ethanwtl.github.io/IBweb/

Related papers

• From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMs [19.943841049221625]
  We introduce the Spatial-Functional Intelligence Benchmark (SFI-Bench), a video-based benchmark with over 1,500 expert-annotated questions.<n>SFI-Bench systematically evaluates two complementary dimensions of advanced reasoning.
  arXiv  Detail & Related papers  (2026-05-04T01:19:36Z)
• Seek-and-Solve: Benchmarking MLLMs for Visual Clue-Driven Reasoning in Daily Scenarios [56.703500104126014]
  DailyClue is a benchmark designed for visual clue-driven reasoning in daily scenarios.<n>Our construction is guided by two core principles: (1) strict grounding in authentic daily activities, and (2) challenging query design.<n>Our analysis reveals several critical insights, emphasizing that the accurate identification of visual clues is essential for robust reasoning.
  arXiv  Detail & Related papers  (2026-04-15T16:22:10Z)
• Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain [62.01012517796797]
  Building robots that can perceive, reason, and act in dynamic, unstructured environments remains a core challenge.<n>Recent embodied systems often adopt a dual-system paradigm, where System 2 handles high-level reasoning while System 1 executes low-level control.<n>We introduce RoboBench, a benchmark that systematically evaluates multimodal large language models (MLLMs) as embodied brains.
  arXiv  Detail & Related papers  (2025-10-20T17:59:03Z)
• What to Ask Next? Probing the Imaginative Reasoning of LLMs with TurtleSoup Puzzles [26.90890466164784]
  We present TurtleSoup-Bench, the first large-scale, bilingual, interactive benchmark for imaginative reasoning.<n>We also propose Mosaic-Agent, a novel agent designed to assess LLMs' performance in this setting.
  arXiv  Detail & Related papers  (2025-08-14T05:55:42Z)
• From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information? [34.959850282872594]
  We present AR-Bench, a novel benchmark designed explicitly to evaluate an LLM's active reasoning skills.<n>AR-Bench comprises three task families-detective cases, situation puzzles, and guessing numbers.<n> Empirical evaluation on AR-Bench demonstrates that contemporary LLMs exhibit pronounced difficulties with active reasoning.
  arXiv  Detail & Related papers  (2025-06-09T23:56:41Z)
• Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluation [106.17986469245302]
  Large language models (LLMs) have demonstrated impressive reasoning capacities that mirror human-like thinking.<n>Existing reasoning benchmarks either focus on domain-specific knowledge (crystallized intelligence) or lack interpretability.<n>We propose DRE-Bench, a dynamic reasoning evaluation benchmark grounded in a hierarchical cognitive framework.
  arXiv  Detail & Related papers  (2025-06-03T09:01:08Z)
• MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models [50.43793764203352]
  We introduce MDK12-Bench, a multi-disciplinary benchmark assessing the reasoning capabilities of MLLMs via real-world K-12 examinations.<n>Our benchmark comprises 140K reasoning instances across diverse difficulty levels from primary school to 12th grade.<n>It features 6,827 instance-level knowledge point annotations based on a well-organized knowledge structure, detailed answer explanations, difficulty labels and cross-year partitions.
  arXiv  Detail & Related papers  (2025-04-08T08:06:53Z)
• MIR-Bench: Can Your LLM Recognize Complicated Patterns via Many-Shot In-Context Reasoning? [21.638848019633595]
  We propose MIR-Bench, the first many-shot in-context reasoning benchmark for pattern recognition.<n>We study many novel problems for many-shot in-context reasoning, and acquired many insightful findings.
  arXiv  Detail & Related papers  (2025-02-14T06:05:12Z)
• EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarios [53.26658545922884]
  We introduce EgoPlan-Bench2, a benchmark designed to assess the planning capabilities of MLLMs across a wide range of real-world scenarios.<n>We evaluate 21 competitive MLLMs and provide an in-depth analysis of their limitations, revealing that they face significant challenges in real-world planning.<n>Our approach enhances the performance of GPT-4V by 10.24 on EgoPlan-Bench2 without additional training.
  arXiv  Detail & Related papers  (2024-12-05T18:57:23Z)
• Multi-Step Reasoning with Large Language Models, a Survey [8.647697652065718]
  This article reviews the field of multi-step reasoning with large language models (LLMs)<n>We propose a taxonomy that identifies different ways to generate, evaluate, and control multi-step reasoning.<n>We find that multi-step reasoning approaches have progressed beyond math word problems, and can now successfully solve challenges in logic, games, and robotics.
  arXiv  Detail & Related papers  (2024-07-16T08:49:35Z)
• Reasoning in Conversation: Solving Subjective Tasks through Dialogue Simulation for Large Language Models [56.93074140619464]
  We propose RiC (Reasoning in Conversation), a method that focuses on solving subjective tasks through dialogue simulation. The motivation of RiC is to mine useful contextual information by simulating dialogues instead of supplying chain-of-thought style rationales. We evaluate both API-based and open-source LLMs including GPT-4, ChatGPT, and OpenChat across twelve tasks.
  arXiv  Detail & Related papers  (2024-02-27T05:37:10Z)
• DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent) [73.10899129264375]
  This paper explores DoraemonGPT, a comprehensive and conceptually elegant system driven by LLMs to understand dynamic scenes.<n>Given a video with a question/task, DoraemonGPT begins by converting the input video into a symbolic memory that stores task-related attributes.<n>We extensively evaluate DoraemonGPT's effectiveness on three benchmarks and several in-the-wild scenarios.
  arXiv  Detail & Related papers  (2024-01-16T14:33:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.