The StatCan Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents

• URL: http://arxiv.org/abs/2304.01412v2
• Date: Wed, 5 Apr 2023 01:20:51 GMT
• Title: The StatCan Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents
• Authors: Xing Han Lu, Siva Reddy, Harm de Vries
• Abstract summary: StatCan Dialogue dataset consists of 19,379 conversation turns between agents working at Statistics Canada and online users looking for published data tables. We propose two tasks: (1) automatic retrieval of relevant tables based on a on-going conversation, and (2) automatic generation of appropriate agent responses at each turn.
• Score: 26.2497150683291
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce the StatCan Dialogue Dataset consisting of 19,379 conversation turns between agents working at Statistics Canada and online users looking for published data tables. The conversations stem from genuine intents, are held in English or French, and lead to agents retrieving one of over 5000 complex data tables. Based on this dataset, we propose two tasks: (1) automatic retrieval of relevant tables based on a on-going conversation, and (2) automatic generation of appropriate agent responses at each turn. We investigate the difficulty of each task by establishing strong baselines. Our experiments on a temporal data split reveal that all models struggle to generalize to future conversations, as we observe a significant drop in performance across both tasks when we move from the validation to the test set. In addition, we find that response generation models struggle to decide when to return a table. Considering that the tasks pose significant challenges to existing models, we encourage the community to develop models for our task, which can be directly used to help knowledge workers find relevant tables for live chat users.

Related papers

• Synthetic Clarification and Correction Dialogues about Data-Centric Tasks -- A Teacher-Student Approach [0.052617184697694476]
  We develop a novel framework for synthetically generating controlled, multi-turn conversations between a user and AI assistant. Each conversation aims to solve a table-based reasoning question through collaborative effort. We employ a strong teacher LLM to verify the correctness of our synthetic conversations.
  arXiv  Detail & Related papers  (2025-03-18T11:37:25Z)
• Exploring Rewriting Approaches for Different Conversational Tasks [63.56404271441824]
  The exact rewriting approach may often depend on the use case and application-specific tasks supported by the conversational assistant. We systematically investigate two different approaches, denoted as rewriting and fusion, on two fundamentally different generation tasks. Our results indicate that the specific rewriting or fusion approach highly depends on the underlying use case and generative task.
  arXiv  Detail & Related papers  (2025-02-26T06:05:29Z)
• Benchmarking Table Comprehension In The Wild [9.224698222634789]
  TableQuest is a new benchmark designed to evaluate the holistic table comprehension capabilities of Large Language Models (LLMs) We experiment with 7 state-of-the-art models, and find that despite reasonable accuracy in locating facts, they often falter when required to execute more sophisticated reasoning or multi-step calculations.
  arXiv  Detail & Related papers  (2024-12-13T05:52:37Z)
• TACT: Advancing Complex Aggregative Reasoning with Information Extraction Tools [51.576974932743596]
  Large Language Models (LLMs) often do not perform well on queries that require the aggregation of information across texts. TACT contains challenging instructions that demand stitching information scattered across one or more texts. We construct this dataset by leveraging an existing dataset of texts and their associated tables. We demonstrate that all contemporary LLMs perform poorly on this dataset, achieving an accuracy below 38%.
  arXiv  Detail & Related papers  (2024-06-05T20:32:56Z)
• TANQ: An open domain dataset of table answered questions [15.323690523538572]
  TANQ is the first open domain question answering dataset where the answers require building tables from information across multiple sources. We release the full source attribution for every cell in the resulting table and benchmark state-of-the-art language models in open, oracle, and closed book setups. Our best-performing baseline, GPT4 reaches an overall F1 score of 29.1, lagging behind human performance by 19.7 points.
  arXiv  Detail & Related papers  (2024-05-13T14:07:20Z)
• Bridging the Gap: Deciphering Tabular Data Using Large Language Model [4.711941969101732]
  This research marks the first application of large language models to table-based question answering tasks. We have architected a distinctive module dedicated to the serialization of tables for seamless integration with expansive language models.
  arXiv  Detail & Related papers  (2023-08-23T03:38:21Z)
• QTSumm: Query-Focused Summarization over Tabular Data [58.62152746690958]
  People primarily consult tables to conduct data analysis or answer specific questions. We define a new query-focused table summarization task, where text generation models have to perform human-like reasoning. We introduce a new benchmark named QTSumm for this task, which contains 7,111 human-annotated query-summary pairs over 2,934 tables.
  arXiv  Detail & Related papers  (2023-05-23T17:43:51Z)
• PRESTO: A Multilingual Dataset for Parsing Realistic Task-Oriented Dialogs [39.58414649004708]
  PRESTO is a dataset of over 550K contextual multilingual conversations between humans and virtual assistants. It contains challenges that occur in real-world NLU tasks such as disfluencies, code-switching, and revisions. Our mT5 model based baselines demonstrate that the conversational phenomenon present in PRESTO are challenging to model.
  arXiv  Detail & Related papers  (2023-03-15T21:51:13Z)
• Weakly Supervised Data Augmentation Through Prompting for Dialogue Understanding [103.94325597273316]
  We present a novel approach that iterates on augmentation quality by applying weakly-supervised filters. We evaluate our methods on the emotion and act classification tasks in DailyDialog and the intent classification task in Facebook Multilingual Task-Oriented Dialogue. For DailyDialog specifically, using 10% of the ground truth data we outperform the current state-of-the-art model which uses 100% of the data.
  arXiv  Detail & Related papers  (2022-10-25T17:01:30Z)
• KETOD: Knowledge-Enriched Task-Oriented Dialogue [77.59814785157877]
  Existing studies in dialogue system research mostly treat task-oriented dialogue and chit-chat as separate domains. We investigate how task-oriented dialogue and knowledge-grounded chit-chat can be effectively integrated into a single model.
  arXiv  Detail & Related papers  (2022-05-11T16:01:03Z)
• Training Conversational Agents with Generative Conversational Networks [74.9941330874663]
  We use Generative Conversational Networks to automatically generate data and train social conversational agents. We evaluate our approach on TopicalChat with automatic metrics and human evaluators, showing that with 10% of seed data it performs close to the baseline that uses 100% of the data.
  arXiv  Detail & Related papers  (2021-10-15T21:46:39Z)
• Simulated Chats for Building Dialog Systems: Learning to Generate Conversations from Instructions [14.47025580681492]
  We present a data creation strategy that uses the pre-trained language model, GPT2, to simulate the interaction between crowd workers by creating a user bot and an agent bot. We demonstrate that by using the simulated data, we achieve significant improvements in low-resource settings on two publicly available datasets.
  arXiv  Detail & Related papers  (2020-10-20T12:04:19Z)
• GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing [117.98107557103877]
  We present GraPPa, an effective pre-training approach for table semantic parsing. We construct synthetic question-pairs over high-free tables via a synchronous context-free grammar. To maintain the model's ability to represent real-world data, we also include masked language modeling.
  arXiv  Detail & Related papers  (2020-09-29T08:17:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.