Improving the Robustness of Summarization Systems with Dual Augmentation
- URL: http://arxiv.org/abs/2306.01090v1
- Date: Thu, 1 Jun 2023 19:04:17 GMT
- Title: Improving the Robustness of Summarization Systems with Dual Augmentation
- Authors: Xiuying Chen, Guodong Long, Chongyang Tao, Mingzhe Li, Xin Gao,
Chengqi Zhang, Xiangliang Zhang
- Abstract summary: A robust summarization system should be able to capture the gist of the document, regardless of the specific word choices or noise in the input.
We first explore the summarization models' robustness against perturbations including word-level synonym substitution and noise.
We propose a SummAttacker, which is an efficient approach to generating adversarial samples based on language models.
- Score: 68.53139002203118
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: A robust summarization system should be able to capture the gist of the
document, regardless of the specific word choices or noise in the input. In
this work, we first explore the summarization models' robustness against
perturbations including word-level synonym substitution and noise. To create
semantic-consistent substitutes, we propose a SummAttacker, which is an
efficient approach to generating adversarial samples based on language models.
Experimental results show that state-of-the-art summarization models have a
significant decrease in performance on adversarial and noisy test sets. Next,
we analyze the vulnerability of the summarization systems and explore improving
the robustness by data augmentation. Specifically, the first brittleness factor
we found is the poor understanding of infrequent words in the input.
Correspondingly, we feed the encoder with more diverse cases created by
SummAttacker in the input space. The other factor is in the latent space, where
the attacked inputs bring more variations to the hidden states. Hence, we
construct adversarial decoder input and devise manifold softmixing operation in
hidden space to introduce more diversity. Experimental results on Gigaword and
CNN/DM datasets demonstrate that our approach achieves significant improvements
over strong baselines and exhibits higher robustness on noisy, attacked, and
clean datasets.
Related papers
- SenTest: Evaluating Robustness of Sentence Encoders [0.4194295877935868]
This work focuses on evaluating the robustness of the sentence encoders.
We employ several adversarial attacks to evaluate its robustness.
The results of the experiments strongly undermine the robustness of sentence encoders.
arXiv Detail & Related papers (2023-11-29T15:21:35Z) - DASA: Difficulty-Aware Semantic Augmentation for Speaker Verification [55.306583814017046]
We present a novel difficulty-aware semantic augmentation (DASA) approach for speaker verification.
DASA generates diversified training samples in speaker embedding space with negligible extra computing cost.
The best result achieves a 14.6% relative reduction in EER metric on CN-Celeb evaluation set.
arXiv Detail & Related papers (2023-10-18T17:07:05Z) - Towards General Visual-Linguistic Face Forgery Detection [95.73987327101143]
Deepfakes are realistic face manipulations that can pose serious threats to security, privacy, and trust.
Existing methods mostly treat this task as binary classification, which uses digital labels or mask signals to train the detection model.
We propose a novel paradigm named Visual-Linguistic Face Forgery Detection(VLFFD), which uses fine-grained sentence-level prompts as the annotation.
arXiv Detail & Related papers (2023-07-31T10:22:33Z) - Improving the Robustness of Summarization Models by Detecting and
Removing Input Noise [50.27105057899601]
We present a large empirical study quantifying the sometimes severe loss in performance from different types of input noise for a range of datasets and model sizes.
We propose a light-weight method for detecting and removing such noise in the input during model inference without requiring any training, auxiliary models, or even prior knowledge of the type of noise.
arXiv Detail & Related papers (2022-12-20T00:33:11Z) - Towards Improving Faithfulness in Abstractive Summarization [37.19777407790153]
We propose a Faithfulness Enhanced Summarization model (FES) to improve fidelity in abstractive summarization.
Our model outperforms strong baselines in experiments on CNN/DM and XSum.
arXiv Detail & Related papers (2022-10-04T19:52:09Z) - Boosting the Discriminant Power of Naive Bayes [17.43377106246301]
We propose a feature augmentation method employing a stack auto-encoder to reduce the noise in the data and boost the discriminant power of naive Bayes.
The experimental results show that the proposed method significantly and consistently outperforms the state-of-the-art naive Bayes classifiers.
arXiv Detail & Related papers (2022-09-20T08:02:54Z) - DoubleMix: Simple Interpolation-Based Data Augmentation for Text
Classification [56.817386699291305]
This paper proposes a simple yet effective data augmentation approach termed DoubleMix.
DoubleMix first generates several perturbed samples for each training data.
It then uses the perturbed data and original data to carry out a two-step in the hidden space of neural models.
arXiv Detail & Related papers (2022-09-12T15:01:04Z) - Bridging the Gap Between Clean Data Training and Real-World Inference
for Spoken Language Understanding [76.89426311082927]
Existing models are trained on clean data, which causes a textitgap between clean data training and real-world inference.
We propose a method from the perspective of domain adaptation, by which both high- and low-quality samples are embedding into similar vector space.
Experiments on the widely-used dataset, Snips, and large scale in-house dataset (10 million training examples) demonstrate that this method not only outperforms the baseline models on real-world (noisy) corpus but also enhances the robustness, that is, it produces high-quality results under a noisy environment.
arXiv Detail & Related papers (2021-04-13T17:54:33Z) - Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven
Cloze Reward [42.925345819778656]
We present ASGARD, a novel framework for Abstractive Summarization with Graph-Augmentation and semantic-driven RewarD.
We propose the use of dual encoders---a sequential document encoder and a graph-structured encoder---to maintain the global context and local characteristics of entities.
Results show that our models produce significantly higher ROUGE scores than a variant without knowledge graph as input on both New York Times and CNN/Daily Mail datasets.
arXiv Detail & Related papers (2020-05-03T18:23:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.