Effectiveness of French Language Models on Abstractive Dialogue Summarization Task

• URL: http://arxiv.org/abs/2207.08305v1
• Date: Sun, 17 Jul 2022 21:43:18 GMT
• Title: Effectiveness of French Language Models on Abstractive Dialogue Summarization Task
• Authors: Yongxin Zhou, Fran\c{c}ois Portet, Fabien Ringeval
• Abstract summary: We present a study on the summarization of spontaneous oral dialogues in French using several language specific pre-trained models. Results show that the BARThez models offer the best performance far above the previous state-of-the-art on DECODA.
• Score: 5.556906034471034
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pre-trained language models have established the state-of-the-art on various natural language processing tasks, including dialogue summarization, which allows the reader to quickly access key information from long conversations in meetings, interviews or phone calls. However, such dialogues are still difficult to handle with current models because the spontaneity of the language involves expressions that are rarely present in the corpora used for pre-training the language models. Moreover, the vast majority of the work accomplished in this field has been focused on English. In this work, we present a study on the summarization of spontaneous oral dialogues in French using several language specific pre-trained models: BARThez, and BelGPT-2, as well as multilingual pre-trained models: mBART, mBARThez, and mT5. Experiments were performed on the DECODA (Call Center) dialogue corpus whose task is to generate abstractive synopses from call center conversations between a caller and one or several agents depending on the situation. Results show that the BARThez models offer the best performance far above the previous state-of-the-art on DECODA. We further discuss the limits of such pre-trained models and the challenges that must be addressed for summarizing spontaneous dialogues.

Related papers

• Large Language Model based Situational Dialogues for Second Language Learning [7.450328495455734]
  In second language learning, scenario-based conversation practice is important for language learners to achieve fluency in speaking. To bridge this gap, we propose situational dialogue models for students to engage in conversational practice. Our situational dialogue models are fine-tuned on large language models (LLMs), with the aim of combining the engaging nature of an open-ended conversation with the focused practice of scenario-based tasks.
  arXiv  Detail & Related papers  (2024-03-29T06:43:55Z)
• SPECTRUM: Speaker-Enhanced Pre-Training for Long Dialogue Summarization [48.284512017469524]
  Multi-turn dialogues are characterized by their extended length and the presence of turn-taking conversations. Traditional language models often overlook the distinct features of these dialogues by treating them as regular text. We propose a speaker-enhanced pre-training method for long dialogue summarization.
  arXiv  Detail & Related papers  (2024-01-31T04:50:00Z)
• FutureTOD: Teaching Future Knowledge to Pre-trained Language Model for Task-Oriented Dialogue [20.79359173822053]
  We propose a novel dialogue pre-training model, FutureTOD, which distills future knowledge to the representation of the previous dialogue context. Our intuition is that a good dialogue representation both learns local context information and predicts future information.
  arXiv  Detail & Related papers  (2023-06-17T10:40:07Z)
• Pre-training Multi-party Dialogue Models with Latent Discourse Inference [85.9683181507206]
  We pre-train a model that understands the discourse structure of multi-party dialogues, namely, to whom each utterance is replying. To fully utilize the unlabeled data, we propose to treat the discourse structures as latent variables, then jointly infer them and pre-train the discourse-aware model.
  arXiv  Detail & Related papers  (2023-05-24T14:06:27Z)
• Stabilized In-Context Learning with Pre-trained Language Models for Few Shot Dialogue State Tracking [57.92608483099916]
  Large pre-trained language models (PLMs) have shown impressive unaided performance across many NLP tasks. For more complex tasks such as dialogue state tracking (DST), designing prompts that reliably convey the desired intent is nontrivial. We introduce a saliency model to limit dialogue text length, allowing us to include more exemplars per query.
  arXiv  Detail & Related papers  (2023-02-12T15:05:10Z)
• Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models [73.11488464916668]
  This study investigates the dynamics of the multilingual pretraining process. We probe checkpoints taken from throughout XLM-R pretraining, using a suite of linguistic tasks. Our analysis shows that the model achieves high in-language performance early on, with lower-level linguistic skills acquired before more complex ones.
  arXiv  Detail & Related papers  (2022-05-24T03:35:00Z)
• Post-Training Dialogue Summarization using Pseudo-Paraphrasing [12.083992819138716]
  We propose to post-train pretrained language models (PLMs) to rephrase from dialogue to narratives. Comprehensive experiments show that our approach significantly improves vanilla PLMs on dialogue summarization.
  arXiv  Detail & Related papers  (2022-04-28T13:42:19Z)
• DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization [19.918194137007653]
  We present a pre-training framework for long dialogue understanding and summarization. Considering the nature of long conversations, we propose a window-based denoising approach for generative pre-training. We conduct extensive experiments on five datasets of long dialogues, covering tasks of dialogue summarization, abstractive question answering and topic segmentation.
  arXiv  Detail & Related papers  (2021-09-06T13:55:03Z)
• Filling the Gap of Utterance-aware and Speaker-aware Representation for Multi-turn Dialogue [76.88174667929665]
  A multi-turn dialogue is composed of multiple utterances from two or more different speaker roles. In the existing retrieval-based multi-turn dialogue modeling, the pre-trained language models (PrLMs) as encoder represent the dialogues coarsely. We propose a novel model to fill such a gap by modeling the effective utterance-aware and speaker-aware representations entailed in a dialogue history.
  arXiv  Detail & Related papers  (2020-09-14T15:07:19Z)
• TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue [113.45485470103762]
  In this work, we unify nine human-human and multi-turn task-oriented dialogue datasets for language modeling. To better model dialogue behavior during pre-training, we incorporate user and system tokens into the masked language modeling.
  arXiv  Detail & Related papers  (2020-04-15T04:09:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.