Can one size fit all?: Measuring Failure in Multi-Document Summarization Domain Transfer

• URL: http://arxiv.org/abs/2503.15768v1
• Date: Thu, 20 Mar 2025 00:57:38 GMT
• Title: Can one size fit all?: Measuring Failure in Multi-Document Summarization Domain Transfer
• Authors: Alexandra DeLucia, Mark Dredze,
• Abstract summary: Multi-document summarization (MDS) is the task of automatically summarizing information in multiple documents.<n>We evaluate MDS models across training approaches, domains, and dimensions.
• Score: 64.03237620355455
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Abstractive multi-document summarization (MDS) is the task of automatically summarizing information in multiple documents, from news articles to conversations with multiple speakers. The training approaches for current MDS models can be grouped into four approaches: end-to-end with special pre-training ("direct"), chunk-then-summarize, extract-then-summarize, and inference with GPT-style models. In this work, we evaluate MDS models across training approaches, domains, and dimensions (reference similarity, quality, and factuality), to analyze how and why models trained on one domain can fail to summarize documents from another (News, Science, and Conversation) in the zero-shot domain transfer setting. We define domain-transfer "failure" as a decrease in factuality, higher deviation from the target, and a general decrease in summary quality. In addition to exploring domain transfer for MDS models, we examine potential issues with applying popular summarization metrics out-of-the-box.

Related papers

• Robust Domain Misinformation Detection via Multi-modal Feature Alignment [49.89164555394584]
  We propose a robust domain and cross-modal approach for multi-modal misinformation detection. It reduces the domain shift by aligning the joint distribution of textual and visual modalities. We also propose a framework that simultaneously considers application scenarios of domain generalization.
  arXiv  Detail & Related papers  (2023-11-24T07:06:16Z)
• Absformer: Transformer-based Model for Unsupervised Multi-Document Abstractive Summarization [1.066048003460524]
  Multi-document summarization (MDS) refers to the task of summarizing the text in multiple documents into a concise summary. Abstractive MDS aims to generate a coherent and fluent summary for multiple documents using natural language generation techniques. We propose Absformer, a new Transformer-based method for unsupervised abstractive summary generation.
  arXiv  Detail & Related papers  (2023-06-07T21:18:23Z)
• Combining State-of-the-Art Models with Maximal Marginal Relevance for Few-Shot and Zero-Shot Multi-Document Summarization [0.6690874707758508]
  Multi-document summarization (MDS) poses many challenges to researchers above those posed by single-document summarization (SDS) We propose a strategy for combining state-of-the-art models' outputs using maximal marginal relevance (MMR) Our MMR-based approach shows improvement over some aspects of the current state-of-the-art results in both few-shot and zero-shot MDS applications.
  arXiv  Detail & Related papers  (2022-11-19T21:46:31Z)
• Disentangled Modeling of Domain and Relevance for Adaptable Dense Retrieval [54.349418995689284]
  We propose a novel Dense Retrieval (DR) framework named Disentangled Dense Retrieval ( DDR) to support effective domain adaptation for DR models. By making the REM and DAMs disentangled, DDR enables a flexible training paradigm in which REM is trained with supervision once and DAMs are trained with unsupervised data. DDR significantly improves ranking performance compared to strong DR baselines and substantially outperforms traditional retrieval methods in most scenarios.
  arXiv  Detail & Related papers  (2022-08-11T11:18:50Z)
• Multi-Document Summarization with Centroid-Based Pretraining [35.8335939654861]
  In Multi-Document Summarization (MDS), the input can be modeled as a set of documents, and the output is its summary. We introduce a novel pretraining objective, which involves selecting the ROUGE-based centroid of each document cluster as a proxy for its summary. Our objective thus does not require human written summaries and can be utilized for pretraining on a dataset consisting solely of document sets.
  arXiv  Detail & Related papers  (2022-08-01T17:28:02Z)
• META: Mimicking Embedding via oThers' Aggregation for Generalizable Person Re-identification [68.39849081353704]
  Domain generalizable (DG) person re-identification (ReID) aims to test across unseen domains without access to the target domain data at training time. This paper presents a new approach called Mimicking Embedding via oThers' Aggregation (META) for DG ReID.
  arXiv  Detail & Related papers  (2021-12-16T08:06:50Z)
• Topic-Guided Abstractive Multi-Document Summarization [21.856615677793243]
  A critical point of multi-document summarization (MDS) is to learn the relations among various documents. We propose a novel abstractive MDS model, in which we represent multiple documents as a heterogeneous graph. We employ a neural topic model to jointly discover latent topics that can act as cross-document semantic units.
  arXiv  Detail & Related papers  (2021-10-21T15:32:30Z)
• Learning to Generalize Unseen Domains via Memory-based Multi-Source Meta-Learning for Person Re-Identification [59.326456778057384]
  We propose the Memory-based Multi-Source Meta-Learning framework to train a generalizable model for unseen domains. We also present a meta batch normalization layer (MetaBN) to diversify meta-test features. Experiments demonstrate that our M$3$L can effectively enhance the generalization ability of the model for unseen domains.
  arXiv  Detail & Related papers  (2020-12-01T11:38:16Z)
• A Review of Single-Source Deep Unsupervised Visual Domain Adaptation [81.07994783143533]
  Large-scale labeled training datasets have enabled deep neural networks to excel across a wide range of benchmark vision tasks. In many applications, it is prohibitively expensive and time-consuming to obtain large quantities of labeled data. To cope with limited labeled training data, many have attempted to directly apply models trained on a large-scale labeled source domain to another sparsely labeled or unlabeled target domain.
  arXiv  Detail & Related papers  (2020-09-01T00:06:50Z)
• Large Margin Mechanism and Pseudo Query Set on Cross-Domain Few-Shot Learning [34.81309962770134]
  Cross-domain few-shot learning is a new branch of few-shot learning problems. We propose a novel large margin fine-tuning method (LMM-PQS) to solve this problem. Our approach is robust and can easily adapt pre-trained models to new domains with few data.
  arXiv  Detail & Related papers  (2020-05-19T05:28:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.