Trapped by Expectations: Functional Fixedness in LLM-Enabled Chat Search

• URL: http://arxiv.org/abs/2504.02074v1
• Date: Wed, 02 Apr 2025 19:14:01 GMT
• Title: Trapped by Expectations: Functional Fixedness in LLM-Enabled Chat Search
• Authors: Jiqun Liu, Jamshed Karimnazarov, Ryen W. White,
• Abstract summary: We investigated the impact of functional fixedness on large language models (LLM)-enabled chat search.<n>We found pre-chat expectations shape language use and prompting behavior.<n>With appropriate system support, this may promote broader exploration of LLM capabilities.
• Score: 9.166043188084414
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Functional fixedness, a cognitive bias that restricts users' interactions with a new system or tool to expected or familiar ways, limits the full potential of Large Language Model (LLM)-enabled chat search, especially in complex and exploratory tasks. To investigate its impact, we conducted a crowdsourcing study with 450 participants, each completing one of six decision-making tasks spanning public safety, diet and health management, sustainability, and AI ethics. Participants engaged in a multi-prompt conversation with ChatGPT to address the task, allowing us to compare pre-chat intent-based expectations with observed interactions. We found that: 1) Several aspects of pre-chat expectations are closely associated with users' prior experiences with ChatGPT, search engines, and virtual assistants; 2) Prior system experience shapes language use and prompting behavior. Frequent ChatGPT users reduced deictic terms and hedge words and frequently adjusted prompts. Users with rich search experience maintained structured, less-conversational queries with minimal modifications. Users of virtual assistants favored directive, command-like prompts, reinforcing functional fixedness; 3) When the system failed to meet expectations, participants generated more detailed prompts with increased linguistic diversity, reflecting adaptive shifts. These findings suggest that while preconceived expectations constrain early interactions, unmet expectations can motivate behavioral adaptation. With appropriate system support, this may promote broader exploration of LLM capabilities. This work also introduces a typology for user intents in chat search and highlights the importance of mitigating functional fixedness to support more creative and analytical use of LLMs.

Related papers

• Understanding Learner-LLM Chatbot Interactions and the Impact of Prompting Guidelines [9.834055425277874]
  This study investigates learner-AI interactions through an educational experiment in which participants receive structured guidance on effective prompting. To assess user behavior and prompting efficacy, we analyze a dataset of 642 interactions from 107 users. Our findings provide a deeper understanding of how users engage with Large Language Models and the role of structured prompting guidance in enhancing AI-assisted communication.
  arXiv  Detail & Related papers  (2025-04-10T15:20:43Z)
• Exploring the Impact of Personality Traits on Conversational Recommender Systems: A Simulation with Large Language Models [70.180385882195]
  This paper introduces a personality-aware user simulation for Conversational Recommender Systems (CRSs) The user agent induces customizable personality traits and preferences, while the system agent possesses the persuasion capability to simulate realistic interaction in CRSs. Experimental results demonstrate that state-of-the-art LLMs can effectively generate diverse user responses aligned with specified personality traits.
  arXiv  Detail & Related papers  (2025-04-09T13:21:17Z)
• Unimib Assistant: designing a student-friendly RAG-based chatbot for all their needs [1.0805849839756092]
  This pilot study focuses on specializing ChatGPT behavior through a Retrieval-Augmented Generation (RAG) system using the OpenAI custom GPTs feature.<n>We created a Unimib Assistant to provide information and solutions to the specific needs of University of Milano-Bicocca (Unimib) students.<n>The satisfaction and overall experience of the users was impaired by the system's inability to always provide fully accurate information.
  arXiv  Detail & Related papers  (2024-11-29T09:07:21Z)
• Navigating the Unknown: A Chat-Based Collaborative Interface for Personalized Exploratory Tasks [35.09558253658275]
  This paper introduces the Collaborative Assistant for Personalized Exploration (CARE) CARE is a system designed to enhance personalization in exploratory tasks by combining a multi-agent LLM framework with a structured user interface. Our findings highlight CARE's potential to transform LLM-based systems from passive information retrievers to proactive partners in personalized problem-solving and exploration.
  arXiv  Detail & Related papers  (2024-10-31T15:30:55Z)
• Modulating Language Model Experiences through Frictions [56.17593192325438]
  Over-consumption of language model outputs risks propagating unchecked errors in the short-term and damaging human capabilities for critical thinking in the long-term. We propose selective frictions for language model experiences, inspired by behavioral science interventions, to dampen misuse.
  arXiv  Detail & Related papers  (2024-06-24T16:31:11Z)
• Beyond ChatBots: ExploreLLM for Structured Thoughts and Personalized Model Responses [35.74453152447319]
  ExploreLLM allows users to structure thoughts, help explore different options, navigate through the choices and recommendations, and to more easily steer models to generate more personalized responses. We conduct a user study and show that users find it helpful to use ExploreLLM for exploratory or planning tasks, because it provides a useful schema-like structure to the task, and guides users in planning. The study also suggests that users can more easily personalize responses with high-level preferences with ExploreLLM.
  arXiv  Detail & Related papers  (2023-12-01T18:31:28Z)
• Multi-Purpose NLP Chatbot : Design, Methodology & Conclusion [0.0]
  This research paper provides a thorough analysis of the chatbots technology environment as it exists today. It provides a very flexible system that makes use of reinforcement learning strategies to improve user interactions and conversational experiences. The complexity of chatbots technology development is also explored in this study, along with the causes that have propelled these developments and their far-reaching effects on a range of sectors.
  arXiv  Detail & Related papers  (2023-10-13T09:47:24Z)
• Leveraging Large Language Models for Automated Dialogue Analysis [12.116834890063146]
  This paper investigates the ability of a state-of-the-art large language model (LLM), ChatGPT-3.5, to perform dialogue behavior detection for nine categories in real human-bot dialogues. Our findings reveal that neither specialized models nor ChatGPT have yet achieved satisfactory results for this task, falling short of human performance.
  arXiv  Detail & Related papers  (2023-09-12T18:03:55Z)
• Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds [59.71218039095155]
  We evaluate language understanding capacities on simple inference tasks that most humans find trivial. We target (i) grammatically-specified entailments, (ii) premises with evidential adverbs of uncertainty, and (iii) monotonicity entailments. The models exhibit moderate to low performance on these evaluation sets.
  arXiv  Detail & Related papers  (2023-05-24T06:41:09Z)
• ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning [70.57126720079971]
  Large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP) This paper evaluates ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium, low, and extremely low resources. Compared to the performance of previous models, our extensive experimental results demonstrate a worse performance of ChatGPT for different NLP tasks and languages.
  arXiv  Detail & Related papers  (2023-04-12T05:08:52Z)
• A Preliminary Evaluation of ChatGPT for Zero-shot Dialogue Understanding [55.37338324658501]
  Zero-shot dialogue understanding aims to enable dialogue to track the user's needs without any training data. In this work, we investigate the understanding ability of ChatGPT for zero-shot dialogue understanding tasks.
  arXiv  Detail & Related papers  (2023-04-09T15:28:36Z)
• Interacting with Non-Cooperative User: A New Paradigm for Proactive Dialogue Policy [83.61404191470126]
  We propose a new solution named I-Pro that can learn Proactive policy in the Interactive setting. Specifically, we learn the trade-off via a learned goal weight, which consists of four factors. The experimental results demonstrate I-Pro significantly outperforms baselines in terms of effectiveness and interpretability.
  arXiv  Detail & Related papers  (2022-04-07T14:11:31Z)
• Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environments [54.405920619915655]
  We introduce Mobile app Tasks with Iterative Feedback (MoTIF), a dataset with natural language commands for the greatest number of interactive environments to date. MoTIF is the first to contain natural language requests for interactive environments that are not satisfiable. We perform initial feasibility classification experiments and only reach an F1 score of 37.3, verifying the need for richer vision-language representations.
  arXiv  Detail & Related papers  (2021-04-17T14:48:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.