GroupTravelBench: Benchmarking LLM Agents on Multi-Person Travel Planning

• URL: http://arxiv.org/abs/2605.25200v1
• Date: Sun, 24 May 2026 17:59:58 GMT
• Title: GroupTravelBench: Benchmarking LLM Agents on Multi-Person Travel Planning
• Abstract summary: We introduce textbfGroupTravelBench, the first benchmark for textbfmulti-user, multi-turn travel planning.<n>Based on real user profiles, POI data, and ticket price data, we synthesize 650 tasks and divide them into three difficulty levels.<n>We evaluate a wide range of LLMs and find that even frontier models still show substantial weaknesses in preference coverage and group fairness.
• Score: 21.321198339300334
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Travel planning is a realistic task for evaluating the planning and tool-use abilities of LLM agents. However, existing benchmarks typically assume only a single user, thereby avoiding one of the most challenging aspects of real-world scenarios: an agent's ability to identify and resolve conflicts among multiple users. To address this gap, we introduce \textbf{GroupTravelBench}, the first benchmark for \textbf{multi-user, multi-turn} travel planning. Based on real user profiles, POI data, and ticket price data, we synthesize 650 tasks and divide them into three difficulty levels. Beyond standard abilities in single-user itinerary planning, such as multi-step reasoning and tool use, our benchmark further evaluates three key capabilities required for travel agents: \emph{(i) elicitation} -- proactively engaging in multi-turn dialogue to gather preferences from each user; \emph{(ii) coordination} -- resolving conflicts among users through compromise or subgrouping strategies; and \emph{(iii) planning} -- searching for travel plans that maximize overall group utility while maintaining fairness and feasibility. To simulate real-world conversational itinerary planning while enabling reliable tool use and offline evaluation, we build an interactive sandbox environment with cached real-world tool data. We evaluate a wide range of LLMs and find that even frontier models still show substantial weaknesses in preference coverage and group fairness. \textit{GroupTravelBench} provides a practical and reproducible benchmark for advancing research on LLM agents for real-world travel planning.

Related papers

• TRIPPULSE: Multi-Agent Travel Planning with Review-Grounded Reasoning [8.15237621583531]
  We propose TRIPPULSE, a multi- agent framework for review-grounded travel planning.<n>We augment TRIPCRAFT with 100K+ real-world reviews and introduce Review-Grounded Per- sona Alignment (RGPA), an LLM-as-a-Judge metric for evaluating alignment with human-centric travel experiences.
  arXiv  Detail & Related papers  (2026-08-31T15:02:37Z)
• Trip+: Benchmarking Agents in Personalized Interactive Travel Planning [12.167681198611744]
  We introduce Trip+ to measure the ability of agents to plan travel holistically.<n>Given traveler profiles and dynamic interactions, agents must generate and revise minute-level itineraries.<n>End-to-end traveler experiences are evaluated via an LLM-based simulator.
  arXiv  Detail & Related papers  (2026-06-19T07:17:40Z)
• MagicAgent: Towards Generalized Agent Planning [73.21129030631421]
  We present textbfMagicAgent, a series of foundation models specifically designed for generalized agent planning.<n>We introduce a lightweight and scalable synthetic data framework that generates high-quality trajectories across diverse planning tasks.<n>We show that MagicAgent-32B and MagicAgent-30B-A3B achieve superior performance across diverse open-source benchmarks.
  arXiv  Detail & Related papers  (2026-02-22T01:39:16Z)
• Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents [52.30603055218294]
  Trajectory2Task is a verifiable data generation pipeline for studying tool use at scale under three realistic user scenarios.<n>It converts valid tool-call trajectories into user-facing tasks with controlled intent adaptations.<n>We benchmark seven state-of-the-art LLMs on the generated complex user scenario tasks and observe frequent failures.
  arXiv  Detail & Related papers  (2026-01-28T00:36:13Z)
• TravelBench: A Broader Real-World Benchmark for Multi-Turn and Tool-Using Travel Planning [22.3041021610283]
  Travel planning is a natural real-world task to test large language models (LLMs) planning and tool-use abilities.<n>TravelBench is a benchmark for fully real-world travel planning.
  arXiv  Detail & Related papers  (2025-12-27T18:25:14Z)
• COMPASS: A Multi-Turn Benchmark for Tool-Mediated Planning & Preference Optimization [47.26757420020116]
  We introduce a benchmark that evaluates agents on realistic travel-planning scenarios.<n>We build a realistic travel database covering transportation, accommodation, and ticketing for 20 U.S. National Parks.<n>We uncover two critical gaps: (i) an acceptable-optimal gap, where agents reliably meet constraints but fail to optimize preferences, and (ii) a plan-coordination gap, where performance collapses on multi-service (flight and hotel) coordination tasks.
  arXiv  Detail & Related papers  (2025-10-08T14:09:46Z)
• ATLAS: Constraints-Aware Multi-Agent Collaboration for Real-World Travel Planning [53.065247112514534]
  ATLAS is a general multi-agent framework designed to handle complex nature of constraints awareness in real-world travel planning tasks.<n>We demonstrate state-of-the-art performance on the TravelPlanner benchmark, improving the final pass rate from 23.3% to 44.4% over its best alternative.
  arXiv  Detail & Related papers  (2025-09-29T23:23:52Z)
• RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing [133.0641538589466]
  RMTBench is a comprehensive textbfuser-centric bilingual role-playing benchmark featuring 80 diverse characters and over 8,000 dialogue rounds.<n>Our benchmark constructs dialogues based on explicit user motivations rather than character descriptions, ensuring alignment with practical user applications.<n>By shifting focus from character background to user intention fulfillment, RMTBench bridges the gap between academic evaluation and practical deployment requirements.
  arXiv  Detail & Related papers  (2025-07-27T16:49:47Z)
• Wide-Horizon Thinking and Simulation-Based Evaluation for Real-World LLM Planning with Multifaceted Constraints [39.01715254437105]
  This paper introduces the Multiple Aspects of Planning (MAoP) to solve planning problems with multifaceted constraints.<n>Instead of direct planning, MAoP leverages the strategist to conduct pre-planning from various aspects and provide the planning blueprint for planners.
  arXiv  Detail & Related papers  (2025-06-14T09:37:59Z)
• FamilyTool: A Multi-hop Personalized Tool Use Benchmark [93.80355496575281]
  FamilyTool is a benchmark grounded in a family-based knowledge graph (KG) that simulates personalized, multi-hop tool use scenarios.<n> Experiments reveal significant performance gaps in state-of-the-art Large Language Models (LLMs)<n>FamilyTool serves as a critical resource for evaluating and advancing LLM agents' reasoning, adaptability, and scalability in complex, dynamic environments.
  arXiv  Detail & Related papers  (2025-04-09T10:42:36Z)
• EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarios [53.26658545922884]
  We introduce EgoPlan-Bench2, a benchmark designed to assess the planning capabilities of MLLMs across a wide range of real-world scenarios.<n>We evaluate 21 competitive MLLMs and provide an in-depth analysis of their limitations, revealing that they face significant challenges in real-world planning.<n>Our approach enhances the performance of GPT-4V by 10.24 on EgoPlan-Bench2 without additional training.
  arXiv  Detail & Related papers  (2024-12-05T18:57:23Z)
• TRIP-PAL: Travel Planning with Guarantees by Combining Large Language Models and Automated Planners [6.378824981027464]
  Traditional approaches rely on problem formulation in a given formal language. Recent Large Language Model (LLM) based approaches directly output plans from user requests using language. We propose TRIP-PAL, a hybrid method that combines the strengths of LLMs and automated planners.
  arXiv  Detail & Related papers  (2024-06-14T17:31:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.