A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Users

• URL: http://arxiv.org/abs/2509.18632v1
• Date: Tue, 23 Sep 2025 04:33:30 GMT
• Title: A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Users
• Authors: Nishant Balepur, Matthew Shu, Yoo Yeon Sung, Seraphina Goldfarb-Tarrant, Shi Feng, Fumeng Yang, Rachel Rudinger, Jordan Lee Boyd-Graber,
• Abstract summary: alignment methods aim to ensure LLM plans are helpful, but train (RLHF) or evaluate (Chatbotorama) on what users prefer.<n>We test this with PlanArena: an interface where 126 users answer 300 multi-step questions with LLM plans.<n>We get 4388 plan executions and 5584 comparisons to measure plan helpfulness (QA success) and user preferences on plans.
• Score: 40.48887496481289
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To assist users in complex tasks, LLMs generate plans: step-by-step instructions towards a goal. While alignment methods aim to ensure LLM plans are helpful, they train (RLHF) or evaluate (ChatbotArena) on what users prefer, assuming this reflects what helps them. We test this with Planorama: an interface where 126 users answer 300 multi-step questions with LLM plans. We get 4388 plan executions and 5584 comparisons to measure plan helpfulness (QA success) and user preferences on plans, and recreate the setup in agents and reward models to see if they simulate or prefer what helps users. We expose: 1) user/model preferences and agent success do not accurately predict which plans help users, so common alignment feedback can misalign with helpfulness; 2) this gap is not due to user-specific preferences, as users are similarly successful when using plans they prefer/disprefer; 3) surface-level cues like brevity and question similarity strongly link to preferences, but such biases fail to predict helpfulness. In all, we argue aligning helpful LLMs needs feedback from real user interactions, not just preferences of what looks helpful, so we discuss the plan NLP researchers can execute to solve this problem.

Related papers

• Toward PDDL Planning Copilot [7.9654550247344895]
  The Planning Copilot integrates multiple planning tools and allows users to invoke them through instructions in natural language.<n>We empirically evaluate the ability of our Planning Copilot to perform planning tasks using three open-source LLMs.<n>Our results show that our Planning Copilot significantly outperforms GPT-5 despite relying on a much smaller LLM.
  arXiv  Detail & Related papers  (2025-09-16T11:51:07Z)
• Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale [51.9706400130481]
  Large Language Models (LLMs) have emerged as personalized assistants for users across a wide range of tasks.<n> PERSONAMEM features curated user profiles with over 180 simulated user-LLM interaction histories.<n>We evaluate LLM chatbots' ability to identify the most suitable response according to the current state of the user's profile.
  arXiv  Detail & Related papers  (2025-04-19T08:16:10Z)
• Hindsight Planner: A Closed-Loop Few-Shot Planner for Embodied Instruction Following [62.10809033451526]
  This work focuses on building a task planner for Embodied Instruction Following (EIF) using Large Language Models (LLMs)<n>We frame the task as a Partially Observable Markov Decision Process (POMDP) and aim to develop a robust planner under a few-shot assumption.<n>Our experiments on the ALFRED dataset indicate that our planner achieves competitive performance under a few-shot assumption.
  arXiv  Detail & Related papers  (2024-12-27T10:05:45Z)
• Query-Efficient Planning with Language Models [8.136901056728945]
  Planning in complex environments requires an agent to efficiently query a world model to find a sequence of actions from start to goal.<n>Recent work has shown that Large Language Models (LLMs) can potentially help with planning by searching over promising states and adapting to feedback from the world.<n>We show that while both approaches improve upon comparable baselines, using an LLM as a generative planner results in significantly fewer interactions.
  arXiv  Detail & Related papers  (2024-12-09T02:51:21Z)
• Towards Efficient LLM Grounding for Embodied Multi-Agent Collaboration [68.29746557968107]
  We propose a novel framework for multi-agent collaboration that introduces Reinforced Advantage feedback (ReAd) for efficient self-refinement of plans.<n> Experiments on Over-AI and a difficult variant of RoCoBench show that ReAd surpasses baselines in success rate, and also significantly decreases the interaction steps of agents.
  arXiv  Detail & Related papers  (2024-05-23T08:33:19Z)
• m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks [31.031053149807857]
  We introduce m&m's: a benchmark containing 4K+ multi-step multi-modal tasks involving 33 tools. For each of these task queries, we provide automatically generated plans using this realistic toolset. We provide a high-quality subset of 1,565 task plans that are human-verified and correctly.
  arXiv  Detail & Related papers  (2024-03-17T04:36:18Z)
• Plan-Grounded Large Language Models for Dual Goal Conversational Settings [7.694972908311347]
  Training Large Language Models to follow user instructions has been shown to supply the LLM with ample capacity to converse fluently while being aligned with humans. Yet, it is not completely clear how an LLM can lead a plan-grounded conversation in mixed-initiative settings. We propose a novel LLM that grounds the dialogue on a procedural plan, can take the dialogue initiative, and enforces guardrails on the system's behavior.
  arXiv  Detail & Related papers  (2024-02-01T22:56:39Z)
• Tree-Planner: Efficient Close-loop Task Planning with Large Language Models [63.06270302774049]
  Tree-Planner reframes task planning with Large Language Models into three distinct phases. Tree-Planner achieves state-of-the-art performance while maintaining high efficiency.
  arXiv  Detail & Related papers  (2023-10-12T17:59:50Z)
• AdaPlanner: Adaptive Planning from Feedback with Language Models [56.367020818139665]
  Large language models (LLMs) have recently demonstrated the potential in acting as autonomous agents for sequential decision-making tasks. We propose a closed-loop approach, AdaPlanner, which allows the LLM agent to refine its self-generated plan adaptively in response to environmental feedback. To mitigate hallucination, we develop a code-style LLM prompt structure that facilitates plan generation across a variety of tasks, environments, and agent capabilities.
  arXiv  Detail & Related papers  (2023-05-26T05:52:27Z)
• Iterative Planning with Plan-Space Explanations: A Tool and User Study [5.779503104475269]
  We implement a tool for human-guided iterative planning including plan-space explanations. The tool runs in standard Web browsers, and provides simple user interfaces for both developers and users. We conduct a first user study, whose outcome indicates the usefulness of plan-property dependency explanations in iterative planning.
  arXiv  Detail & Related papers  (2020-11-19T08:15:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.