BASES: Large-scale Web Search User Simulation with Large Language Model
based Agents
- URL: http://arxiv.org/abs/2402.17505v1
- Date: Tue, 27 Feb 2024 13:44:09 GMT
- Title: BASES: Large-scale Web Search User Simulation with Large Language Model
based Agents
- Authors: Ruiyang Ren, Peng Qiu, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Hua Wu,
Ji-Rong Wen, Haifeng Wang
- Abstract summary: BASES is a novel user simulation framework with large language models (LLMs)
Our simulation framework can generate unique user profiles at scale, which subsequently leads to diverse search behaviors.
WARRIORS is a new large-scale dataset encompassing web search user behaviors, including both Chinese and English versions.
- Score: 108.97507653131917
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Due to the excellent capacities of large language models (LLMs), it becomes
feasible to develop LLM-based agents for reliable user simulation. Considering
the scarcity and limit (e.g., privacy issues) of real user data, in this paper,
we conduct large-scale user simulation for web search, to improve the analysis
and modeling of user search behavior. Specially, we propose BASES, a novel user
simulation framework with LLM-based agents, designed to facilitate
comprehensive simulations of web search user behaviors. Our simulation
framework can generate unique user profiles at scale, which subsequently leads
to diverse search behaviors. To demonstrate the effectiveness of BASES, we
conduct evaluation experiments based on two human benchmarks in both Chinese
and English, demonstrating that BASES can effectively simulate large-scale
human-like search behaviors. To further accommodate the research on web search,
we develop WARRIORS, a new large-scale dataset encompassing web search user
behaviors, including both Chinese and English versions, which can greatly
bolster research in the field of information retrieval. Our code and data will
be publicly released soon.
Related papers
- LLM experiments with simulation: Large Language Model Multi-Agent System for Simulation Model Parametrization in Digital Twins [4.773175285216063]
This paper presents a novel framework that applies large language models (LLMs) to automate the parametrization of simulation models in digital twins.
The proposed approach enhances the usability of simulation model by infusing it with knowledges from LLM.
The system has the potential to increase user-friendliness and reduce the cognitive load on human users.
arXiv Detail & Related papers (2024-05-28T11:59:40Z) - Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study [61.64685376882383]
Counterfactual learning to rank (CLTR) has attracted extensive attention in the IR community for its ability to leverage massive logged user interaction data to train ranking models.
This paper investigates the robustness of existing CLTR models in complex and diverse situations.
We find that the DLA models and IPS-DCM show better robustness under various simulation settings than IPS-PBM and PRS with offline propensity estimation.
arXiv Detail & Related papers (2024-04-04T10:54:38Z) - How Reliable is Your Simulator? Analysis on the Limitations of Current LLM-based User Simulators for Conversational Recommendation [14.646529557978512]
We analyze the limitations of using Large Language Models in constructing user simulators for Conversational Recommender System.
Data leakage, which occurs in conversational history and the user simulator's replies, results in inflated evaluation results.
We propose SimpleUserSim, employing a straightforward strategy to guide the topic toward the target items.
arXiv Detail & Related papers (2024-03-25T04:21:06Z) - USimAgent: Large Language Models for Simulating Search Users [33.17004578463697]
Large Language Models (LLMs) have demonstrated remarked potential in simulating human-level intelligence.
In this paper, we introduce a LLM-based user search behavior simulator, USimAgent.
The proposed simulator can simulate users' querying, clicking, and stopping behaviors during search, and thus, is capable of generating complete search sessions.
arXiv Detail & Related papers (2024-03-14T07:40:54Z) - Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval [56.65147231836708]
We develop SWIM-IR, a synthetic retrieval training dataset containing 33 languages for fine-tuning multilingual dense retrievers.
SAP assists the large language model (LLM) in generating informative queries in the target language.
Our models, called SWIM-X, are competitive with human-supervised dense retrieval models.
arXiv Detail & Related papers (2023-11-10T00:17:10Z) - Unlocking the Potential of User Feedback: Leveraging Large Language
Model as User Simulator to Enhance Dialogue System [65.93577256431125]
We propose an alternative approach called User-Guided Response Optimization (UGRO) to combine it with a smaller task-oriented dialogue model.
This approach uses LLM as annotation-free user simulator to assess dialogue responses, combining them with smaller fine-tuned end-to-end TOD models.
Our approach outperforms previous state-of-the-art (SOTA) results.
arXiv Detail & Related papers (2023-06-16T13:04:56Z) - User Behavior Simulation with Large Language Model based Agents [116.74368915420065]
We propose an LLM-based agent framework and design a sandbox environment to simulate real user behaviors.
Based on extensive experiments, we find that the simulated behaviors of our method are very close to the ones of real humans.
arXiv Detail & Related papers (2023-06-05T02:58:35Z) - Improving Classifier Training Efficiency for Automatic Cyberbullying
Detection with Feature Density [58.64907136562178]
We study the effectiveness of Feature Density (FD) using different linguistically-backed feature preprocessing methods.
We hypothesise that estimating dataset complexity allows for the reduction of the number of required experiments.
The difference in linguistic complexity of datasets allows us to additionally discuss the efficacy of linguistically-backed word preprocessing.
arXiv Detail & Related papers (2021-11-02T15:48:28Z) - Imitate TheWorld: A Search Engine Simulation Platform [13.011052642314421]
We build a simulated search engine AESim that can properly give feedback by a well-trained discriminator for generated pages.
Different from previous simulation platforms which lose connection with the real world, ours depends on the real data in Search.
Our experiments also show AESim can better reflect the online performance of ranking models than classic ranking metrics.
arXiv Detail & Related papers (2021-07-16T03:55:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.