ProductAgent: Benchmarking Conversational Product Search Agent with Asking Clarification Questions

• URL: http://arxiv.org/abs/2407.00942v1
• Date: Mon, 1 Jul 2024 03:50:23 GMT
• Title: ProductAgent: Benchmarking Conversational Product Search Agent with Asking Clarification Questions
• Authors: Jingheng Ye, Yong Jiang, Xiaobin Wang, Yinghui Li, Yangning Li, Hai-Tao Zheng, Pengjun Xie, Fei Huang,
• Abstract summary: ProductAgent is a conversational information seeking agent equipped with abilities of strategic clarification question generation and dynamic product retrieval. We develop the agent with strategies for product feature summarization, query generation, and product retrieval. Experiments show that ProductAgent interacts positively with the user and enhances retrieval performance with increasing dialogue turns.
• Score: 68.81939215223818
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces the task of product demand clarification within an e-commercial scenario, where the user commences the conversation with ambiguous queries and the task-oriented agent is designed to achieve more accurate and tailored product searching by asking clarification questions. To address this task, we propose ProductAgent, a conversational information seeking agent equipped with abilities of strategic clarification question generation and dynamic product retrieval. Specifically, we develop the agent with strategies for product feature summarization, query generation, and product retrieval. Furthermore, we propose the benchmark called PROCLARE to evaluate the agent's performance both automatically and qualitatively with the aid of a LLM-driven user simulator. Experiments show that ProductAgent interacts positively with the user and enhances retrieval performance with increasing dialogue turns, where user demands become gradually more explicit and detailed. All the source codes will be released after the review anonymity period.

Related papers

• ClarQ-LLM: A Benchmark for Models Clarifying and Requesting Information in Task-Oriented Dialog [11.585398152713505]
  We introduce ClarQ-LLM, an evaluation framework consisting of bilingual English-Chinese conversation tasks, conversational agents and evaluation metrics. The benchmark includes 31 different task types, each with 10 unique dialogue scenarios between information seeker and provider agents. Unlike traditional benchmarks that evaluate agents based on fixed dialogue content, ClarQ-LLM includes a provider conversational agent to replicate the original human provider.
  arXiv  Detail & Related papers  (2024-09-09T22:29:35Z)
• Ask-before-Plan: Proactive Language Agents for Real-World Planning [68.08024918064503]
  Proactive Agent Planning requires language agents to predict clarification needs based on user-agent conversation and agent-environment interaction. We propose a novel multi-agent framework, Clarification-Execution-Planning (textttCEP), which consists of three agents specialized in clarification, execution, and planning.
  arXiv  Detail & Related papers  (2024-06-18T14:07:28Z)
• A Conceptual Framework for Conversational Search and Recommendation: Conceptualizing Agent-Human Interactions During the Conversational Search Process [3.2114882156161824]
  The conversational search task aims to enable a user to resolve information needs via natural language dialogue with an agent. We aim to develop a conceptual framework of the actions and intents of users and agents explaining how these actions enable the user to explore the search space and resolve their information need.
  arXiv  Detail & Related papers  (2024-04-12T17:48:18Z)
• Elicitron: An LLM Agent-Based Simulation Framework for Design Requirements Elicitation [38.98478510165569]
  This paper introduces a novel framework that leverages Large Language Models (LLMs) to automate and enhance the requirements elicitation process. LLMs are used to generate a vast array of simulated users (LLM agents), enabling the exploration of a much broader range of user needs.
  arXiv  Detail & Related papers  (2024-04-04T17:36:29Z)
• Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents [110.25679611755962]
  Current language model-driven agents often lack mechanisms for effective user participation, which is crucial given the vagueness commonly found in user instructions. We introduce Intention-in-Interaction (IN3), a novel benchmark designed to inspect users' implicit intentions through explicit queries. We empirically train Mistral-Interact, a powerful model that proactively assesses task vagueness, inquires user intentions, and refines them into actionable goals.
  arXiv  Detail & Related papers  (2024-02-14T14:36:30Z)
• InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks [84.7788065721689]
  In this paper, we introduce InfiAgent-DABench, the first benchmark specifically designed to evaluate LLM-based agents on data analysis tasks. This benchmark contains DAEval, a dataset consisting of 257 data analysis questions derived from 52 CSV files. Building on top of our agent framework, we develop a specialized agent, DAAgent, which surpasses GPT-3.5 by 3.9% on DABench.
  arXiv  Detail & Related papers  (2024-01-10T19:04:00Z)
• An Interactive Query Generation Assistant using LLM-based Prompt Modification and User Feedback [9.461978375200102]
  The proposed interface is a novel search interface which supports automatic and interactive query generation over a mono-linguial or multi-lingual document collection. The interface enables the users to refine the queries generated by different LLMs, to provide feedback on the retrieved documents or passages, and is able to incorporate the users' feedback as prompts to generate more effective queries.
  arXiv  Detail & Related papers  (2023-11-19T04:42:24Z)
• Social Commonsense-Guided Search Query Generation for Open-Domain Knowledge-Powered Conversations [66.16863141262506]
  We present a novel approach that focuses on generating internet search queries guided by social commonsense. Our proposed framework addresses passive user interactions by integrating topic tracking, commonsense response generation and instruction-driven query generation.
  arXiv  Detail & Related papers  (2023-10-22T16:14:56Z)
• Search-Engine-augmented Dialogue Response Generation with Cheaply Supervised Query Production [98.98161995555485]
  We propose a dialogue model that can access the vast and dynamic information from any search engine for response generation. As the core module, a query producer is used to generate queries from a dialogue context to interact with a search engine. Experiments show that our query producer can achieve R@1 and R@5 rates of 62.4% and 74.8% for retrieving gold knowledge.
  arXiv  Detail & Related papers  (2023-02-16T01:58:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.