Exploring Human-Like Thinking in Search Simulations with Large Language Models

• URL: http://arxiv.org/abs/2504.07570v1
• Date: Thu, 10 Apr 2025 09:04:58 GMT
• Title: Exploring Human-Like Thinking in Search Simulations with Large Language Models
• Authors: Erhan Zhang, Xingzhu Wang, Peiyuan Gong, Zixuan Yang, Jiaxin Mao,
• Abstract summary: Simulating user search behavior is a critical task in information retrieval.<n>Recent advancements in large language models (LLMs) have opened up new possibilities for generating human-like actions.<n>We explore the integration of human-like thinking into search simulations by leveraging LLMs to simulate users' hidden cognitive processes.
• Score: 9.825091149361208
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Simulating user search behavior is a critical task in information retrieval, which can be employed for user behavior modeling, data augmentation, and system evaluation. Recent advancements in large language models (LLMs) have opened up new possibilities for generating human-like actions including querying, browsing, and clicking. In this work, we explore the integration of human-like thinking into search simulations by leveraging LLMs to simulate users' hidden cognitive processes. Specifically, given a search task and context, we prompt LLMs to first think like a human before executing the corresponding action. As existing search datasets do not include users' thought processes, we conducted a user study to collect a new dataset enriched with users' explicit thinking. We investigate the impact of incorporating such human-like thinking on simulation performance and apply supervised fine-tuning (SFT) to teach LLMs to emulate both human thinking and actions. Our experiments span two dimensions in leveraging LLMs for user simulation: (1) with or without explicit thinking, and (2) with or without fine-tuning on the thinking-augmented dataset. The results demonstrate the feasibility and potential of incorporating human-like thinking in user simulations, though performance improvements on some metrics remain modest. We believe this exploration provides new avenues and inspirations for advancing user behavior modeling in search simulations.

Related papers

• Can LLMs Simulate Personas with Reversed Performance? A Benchmark for Counterfactual Instruction Following [12.668201122427101]
  Large Language Models (LLMs) are increasingly widely used to simulate personas in virtual environments. We show that even state-of-the-art LLMs cannot simulate personas with reversed performance.
  arXiv  Detail & Related papers  (2025-04-08T22:00:32Z)
• LLM-assisted Explicit and Implicit Multi-interest Learning Framework for Sequential Recommendation [50.98046887582194]
  We propose an explicit and implicit multi-interest learning framework to model user interests on two levels: behavior and semantics. The proposed EIMF framework effectively and efficiently combines small models with LLM to improve the accuracy of multi-interest modeling.
  arXiv  Detail & Related papers  (2024-11-14T13:00:23Z)
• USimAgent: Large Language Models for Simulating Search Users [33.17004578463697]
  We introduce a Large Language Models-based user search behavior simulator, USimAgent. The simulator can simulate users' querying, clicking, and stopping behaviors during search. Empirical investigation on a real user behavior dataset shows that the simulator outperforms existing methods in query generation.
  arXiv  Detail & Related papers  (2024-03-14T07:40:54Z)
• Human Simulacra: Benchmarking the Personification of Large Language Models [38.21708264569801]
  Large language models (LLMs) are recognized as systems that closely mimic aspects of human intelligence.<n>This paper introduces a framework for constructing virtual characters' life stories from the ground up.<n> Experimental results demonstrate that our constructed simulacra can produce personified responses that align with their target characters.
  arXiv  Detail & Related papers  (2024-02-28T09:11:14Z)
• From Summary to Action: Enhancing Large Language Models for Complex Tasks with Open World APIs [62.496139001509114]
  We introduce a novel tool invocation pipeline designed to control massive real-world APIs. This pipeline mirrors the human task-solving process, addressing complicated real-life user queries. Empirical evaluations of our Sum2Act pipeline on the ToolBench benchmark show significant performance improvements.
  arXiv  Detail & Related papers  (2024-02-28T08:42:23Z)
• BASES: Large-scale Web Search User Simulation with Large Language Model based Agents [108.97507653131917]
  BASES is a novel user simulation framework with large language models (LLMs) Our simulation framework can generate unique user profiles at scale, which subsequently leads to diverse search behaviors. WARRIORS is a new large-scale dataset encompassing web search user behaviors, including both Chinese and English versions.
  arXiv  Detail & Related papers  (2024-02-27T13:44:09Z)
• Character-LLM: A Trainable Agent for Role-Playing [67.35139167985008]
  Large language models (LLMs) can be used to serve as agents to simulate human behaviors. We introduce Character-LLM that teach LLMs to act as specific people such as Beethoven, Queen Cleopatra, Julius Caesar, etc.
  arXiv  Detail & Related papers  (2023-10-16T07:58:56Z)
• User Behavior Simulation with Large Language Model based Agents [116.74368915420065]
  We propose an LLM-based agent framework and design a sandbox environment to simulate real user behaviors. Based on extensive experiments, we find that the simulated behaviors of our method are very close to the ones of real humans.
  arXiv  Detail & Related papers  (2023-06-05T02:58:35Z)
• Synthetic Data-Based Simulators for Recommender Systems: A Survey [55.60116686945561]
  This survey aims at providing a comprehensive overview of the recent trends in the field of modeling and simulation. We start with the motivation behind the development of frameworks implementing the simulations -- simulators. We provide a new consistent classification of existing simulators based on their functionality, approbation, and industrial effectiveness.
  arXiv  Detail & Related papers  (2022-06-22T19:33:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.