Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations

• URL: http://arxiv.org/abs/2408.15232v2
• Date: Thu, 17 Oct 2024 20:43:22 GMT
• Title: Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations
• Authors: Yucheng Jiang, Yijia Shao, Dekun Ma, Sina J. Semnani, Monica S. Lam,
• Abstract summary: Collaborative STORM lets users observe and steer the discourse among several LM agents. The agents ask questions on the user's behalf, allowing the user to discover unknown unknowns serendipitously. For automatic evaluation, we construct the WildSeek dataset by collecting real information-seeking records with user goals.
• Score: 8.848859080368799
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While language model (LM)-powered chatbots and generative search engines excel at answering concrete queries, discovering information in the terrain of unknown unknowns remains challenging for users. To emulate the common educational scenario where children/students learn by listening to and participating in conversations of their parents/teachers, we create Collaborative STORM (Co-STORM). Unlike QA systems that require users to ask all the questions, Co-STORM lets users observe and occasionally steer the discourse among several LM agents. The agents ask questions on the user's behalf, allowing the user to discover unknown unknowns serendipitously. To facilitate user interaction, Co-STORM assists users in tracking the discourse by organizing the uncovered information into a dynamic mind map, ultimately generating a comprehensive report as takeaways. For automatic evaluation, we construct the WildSeek dataset by collecting real information-seeking records with user goals. Co-STORM outperforms baseline methods on both discourse trace and report quality. In a further human evaluation, 70% of participants prefer Co-STORM over a search engine, and 78% favor it over a RAG chatbot.

Related papers

• Exploring Knowledge Tracing in Tutor-Student Dialogues [53.52699766206808]
  We present a first attempt at performing knowledge tracing (KT) in tutor-student dialogues. We propose methods to identify the knowledge components/skills involved in each dialogue turn. We then apply a range of KT methods on the resulting labeled data to track student knowledge levels over an entire dialogue.
  arXiv  Detail & Related papers  (2024-09-24T22:31:39Z)
• Improving Ontology Requirements Engineering with OntoChat and Participatory Prompting [3.3241053483599563]
  ORE has primarily relied on manual methods, such as interviews and collaborative forums, to gather user requirements from domain experts. Current OntoChat offers a framework for ORE that utilise large language models (LLMs) to streamline the process. This study produces pre-defined prompt templates based on user queries, focusing on creating and refining personas, goals, scenarios, sample data, and data resources for user stories.
  arXiv  Detail & Related papers  (2024-08-09T19:21:14Z)
• Using LLMs to Investigate Correlations of Conversational Follow-up Queries with User Satisfaction [32.32365329050603]
  We propose a taxonomy of 18 users' follow-up query patterns from conversational search engine Naver Cue:. Compared to the existing literature on query reformulations, we uncovered a new set of motivations and actions behind follow-up queries. Our initial findings suggest some signals of dissatisfactions, such as Clarifying Queries, Excluding Condition, and Substituting Condition with follow-up queries.
  arXiv  Detail & Related papers  (2024-07-18T05:10:35Z)
• Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestion [16.563311988191636]
  We construct an entity-centric knowledge store for each user based on their search and browsing activities on the web. This knowledge store is light-weight, since it only produces user-specific aggregate projections of interests and knowledge onto public knowledge graphs.
  arXiv  Detail & Related papers  (2023-11-10T01:18:47Z)
• AutoConv: Automatically Generating Information-seeking Conversations with Large Language Models [74.10293412011455]
  We propose AutoConv for synthetic conversation generation. Specifically, we formulate the conversation generation problem as a language modeling task. We finetune an LLM with a few human conversations to capture the characteristics of the information-seeking process.
  arXiv  Detail & Related papers  (2023-08-12T08:52:40Z)
• Search-Engine-augmented Dialogue Response Generation with Cheaply Supervised Query Production [98.98161995555485]
  We propose a dialogue model that can access the vast and dynamic information from any search engine for response generation. As the core module, a query producer is used to generate queries from a dialogue context to interact with a search engine. Experiments show that our query producer can achieve R@1 and R@5 rates of 62.4% and 74.8% for retrieving gold knowledge.
  arXiv  Detail & Related papers  (2023-02-16T01:58:10Z)
• INSCIT: Information-Seeking Conversations with Mixed-Initiative Interactions [47.90088587508672]
  InSCIt is a dataset for Information-Seeking Conversations with mixed-initiative Interactions. It contains 4.7K user-agent turns from 805 human-human conversations. We report results of two systems based on state-of-the-art models of conversational knowledge identification and open-domain question answering.
  arXiv  Detail & Related papers  (2022-07-02T06:18:12Z)
• KETOD: Knowledge-Enriched Task-Oriented Dialogue [77.59814785157877]
  Existing studies in dialogue system research mostly treat task-oriented dialogue and chit-chat as separate domains. We investigate how task-oriented dialogue and knowledge-grounded chit-chat can be effectively integrated into a single model.
  arXiv  Detail & Related papers  (2022-05-11T16:01:03Z)
• Information Seeking in the Spirit of Learning: a Dataset for Conversational Curiosity [10.409312809724458]
  We design a Wizard-of-Oz dialog task that tests the hypothesis that engagement increases when users are presented with facts related to what they know. We collect and release 14K dialogs (181K utterances) where users and assistants converse about geographic topics. This dataset is annotated with pre-existing user knowledge, message-level dialog acts, grounding to Wikipedia, and user reactions to messages.
  arXiv  Detail & Related papers  (2020-05-01T01:55:09Z)
• Conversations with Search Engines: SERP-based Conversational Response Generation [77.1381159789032]
  We create a suitable dataset, the Search as a Conversation (SaaC) dataset, for the development of pipelines for conversations with search engines. We also develop a state-of-the-art pipeline for conversations with search engines, the Conversations with Search Engines (CaSE) using this dataset. CaSE enhances the state-of-the-art by introducing a supporting token identification module and aprior-aware pointer generator.
  arXiv  Detail & Related papers  (2020-04-29T13:07:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.