ChatShop: Interactive Information Seeking with Language Agents

• URL: http://arxiv.org/abs/2404.09911v2
• Date: Sun, 16 Jun 2024 15:24:12 GMT
• Title: ChatShop: Interactive Information Seeking with Language Agents
• Authors: Sanxing Chen, Sam Wiseman, Bhuwan Dhingra,
• Abstract summary: desire and ability to seek new information strategically are fundamental to human learning. We analyze a popular web shopping task designed to test language agents' ability to perform strategic exploration. We show that the proposed task can effectively evaluate the agent's ability to explore and gradually accumulate information.
• Score: 16.879814917881895
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The desire and ability to seek new information strategically are fundamental to human learning but often overlooked in current language agent evaluation. We analyze a popular web shopping task designed to test language agents' ability to perform strategic exploration and discover that it can be reformulated and solved as a single-turn retrieval task without the need for interactive information seeking. This finding encourages us to rethink realistic constraints on information access that would necessitate strategic information seeking. We then redesign the task to introduce a notion of task ambiguity and the role of a shopper, serving as a dynamic party with whom the agent strategically interacts in an open-ended conversation to make informed decisions. Our experiments demonstrate that the proposed task can effectively evaluate the agent's ability to explore and gradually accumulate information through multi-turn interactions. Additionally, we show that large language model-simulated shoppers serve as a good proxy for real human shoppers, revealing similar error patterns in agents.

Related papers

• A Survey on Complex Tasks for Goal-Directed Interactive Agents [60.53915548970061]
  This survey compiles relevant tasks and environments for evaluating goal-directed interactive agents. An up-to-date compilation of relevant resources can be found on our project website.
  arXiv  Detail & Related papers  (2024-09-27T08:17:53Z)
• ProductAgent: Benchmarking Conversational Product Search Agent with Asking Clarification Questions [68.81939215223818]
  ProductAgent is a conversational information seeking agent equipped with abilities of strategic clarification question generation and dynamic product retrieval. We develop the agent with strategies for product feature summarization, query generation, and product retrieval. Experiments show that ProductAgent interacts positively with the user and enhances retrieval performance with increasing dialogue turns.
  arXiv  Detail & Related papers  (2024-07-01T03:50:23Z)
• Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents [110.25679611755962]
  Current language model-driven agents often lack mechanisms for effective user participation, which is crucial given the vagueness commonly found in user instructions. We introduce Intention-in-Interaction (IN3), a novel benchmark designed to inspect users' implicit intentions through explicit queries. We empirically train Mistral-Interact, a powerful model that proactively assesses task vagueness, inquires user intentions, and refines them into actionable goals.
  arXiv  Detail & Related papers  (2024-02-14T14:36:30Z)
• Investigating Reinforcement Learning for Communication Strategies in a Task-Initiative Setting [8.680676599607123]
  We analyze the trade-offs between initial presentation and subsequent followup as a function of user clarification strategy. We find surprising advantages to coherence-based representations of dialogue strategy, which bring minimal data requirements, explainable choices, and strong audit capabilities.
  arXiv  Detail & Related papers  (2023-08-03T00:10:23Z)
• CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society [58.04479313658851]
  This paper explores the potential of building scalable techniques to facilitate autonomous cooperation among communicative agents. We propose a novel communicative agent framework named role-playing. Our contributions include introducing a novel communicative agent framework, offering a scalable approach for studying the cooperative behaviors and capabilities of multi-agent systems.
  arXiv  Detail & Related papers  (2023-03-31T01:09:00Z)
• Improving Language Model Prompting in Support of Semi-autonomous Task Learning [6.021787236982658]
  Language models (LLMs) offer potential as a source of knowledge for agents that need to acquire new task competencies. We describe efforts toward a novel agent capability that can construct cues that result in useful LLM responses for an agent learning a new task.
  arXiv  Detail & Related papers  (2022-09-13T15:36:01Z)
• Learning When and What to Ask: a Hierarchical Reinforcement Learning Framework [17.017688226277834]
  We formulate a hierarchical reinforcement learning framework for learning to decide when to request additional information from humans. Results on a simulated human-assisted navigation problem demonstrate the effectiveness of our framework.
  arXiv  Detail & Related papers  (2021-10-14T01:30:36Z)
• Boosting Search Engines with Interactive Agents [25.89284695491093]
  This paper presents first steps in designing agents that learn meta-strategies for contextual query refinements. Agents are empowered with simple but effective search operators to exert fine-grained and transparent control over queries and search results.
  arXiv  Detail & Related papers  (2021-09-01T13:11:57Z)
• Can You be More Social? Injecting Politeness and Positivity into Task-Oriented Conversational Agents [60.27066549589362]
  Social language used by human agents is associated with greater users' responsiveness and task completion. The model uses a sequence-to-sequence deep learning architecture, extended with a social language understanding element. Evaluation in terms of content preservation and social language level using both human judgment and automatic linguistic measures shows that the model can generate responses that enable agents to address users' issues in a more socially appropriate way.
  arXiv  Detail & Related papers  (2020-12-29T08:22:48Z)
• SPA: Verbal Interactions between Agents and Avatars in Shared Virtual Environments using Propositional Planning [61.335252950832256]
  Sense-Plan-Ask, or SPA, generates plausible verbal interactions between virtual human-like agents and user avatars in shared virtual environments. We find that our algorithm creates a small runtime cost and enables agents to complete their goals more effectively than agents without the ability to leverage natural-language communication.
  arXiv  Detail & Related papers  (2020-02-08T23:15:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.