Unsupervised Summarization Re-ranking

• URL: http://arxiv.org/abs/2212.09593v4
• Date: Thu, 14 Nov 2024 06:00:39 GMT
• Title: Unsupervised Summarization Re-ranking
• Abstract summary: In this paper, we propose to re-rank summary candidates in an unsupervised manner, aiming to close the performance gap between unsupervised and supervised models. Our approach improves the unsupervised PEG by up to 7.27% and ChatGPT by up to 6.86% relative mean ROUGE across four widely-adopted summarization benchmarks.
• Score: 27.8646077587359
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rise of task-specific pre-training objectives, abstractive summarization models like PEGASUS offer appealing zero-shot performance on downstream summarization tasks. However, the performance of such unsupervised models still lags significantly behind their supervised counterparts. Similarly to the supervised setup, we notice a very high variance in quality among summary candidates from these models while only one candidate is kept as the summary output. In this paper, we propose to re-rank summary candidates in an unsupervised manner, aiming to close the performance gap between unsupervised and supervised models. Our approach improves the unsupervised PEGASUS by up to 7.27% and ChatGPT by up to 6.86% relative mean ROUGE across four widely-adopted summarization benchmarks ; and achieves relative gains of 7.51% (up to 23.73% from XSum to WikiHow) averaged over 30 zero-shot transfer setups (finetuning on a dataset, evaluating on another).

Related papers

• The Capability Frontier: Benchmarks Miss 82% of Model Performance [7.258179693117413]
  Existing benchmarks typically report accuracy for a single model on a single run.<n>This systematically understates real-world LLM capabilities, particularly under heterogeneous data distributions.
  arXiv  Detail & Related papers  (2026-06-25T10:20:47Z)
• PGDG: Physically Grounded Data Generation for Robust Bimanual Policy Learning from a Single Demonstration [13.432047023375608]
  We propose PGDG, a data generation framework with zero-shot curation.<n>PGDG iterates between a physics-grounded sampler and a dataset curator.<n>It consistently outperforms spatial-only augmentation in both simulation and real-world transfer.
  arXiv  Detail & Related papers  (2026-05-20T20:14:24Z)
• Efficient Benchmarking of AI Agents [0.0]
  We study whether small task subsets can preserve agent rankings at substantially lower cost.<n>We find that absolute score prediction degrades under scaffold-driven distribution shift.<n>We propose an optimization-free protocol: evaluate new agents only on tasks with intermediate historical pass rates.
  arXiv  Detail & Related papers  (2026-03-24T22:17:11Z)
• MACEval: A Multi-Agent Continual Evaluation Network for Large Models [52.629762680215315]
  We introduce MACEval, a Multi-Agent Continual Evaluation network for dynamic evaluation of large models.<n>We demonstrate that MACEval is (1) human-free and automatic, mitigating laborious result processing with inter-agent judgment guided; (2) efficient and economical, reducing a considerable amount of data and overhead to obtain similar results compared to related benchmarks; and (3) flexible and scalable, migrating or integrating existing benchmarks via customized evaluation topologies.
  arXiv  Detail & Related papers  (2025-11-12T09:26:24Z)
• Pretrained Reversible Generation as Unsupervised Visual Representation Learning [7.076111872035312]
  We propose Pretrained Reversible Generation (PRG), which extracts unsupervised representations by reversing the generative process of a pretrained continuous generation model. PRG effectively reuses unsupervised generative models, leveraging their high capacity to serve as robust and generalizable feature extractors for downstream tasks. Our method consistently outperforms prior approaches across multiple benchmarks, achieving state-of-the-art performance among generative model based methods, including 78% top-1 accuracy on ImageNet at a resolution of 64.
  arXiv  Detail & Related papers  (2024-11-29T08:24:49Z)
• STOP! Benchmarking Large Language Models with Sensitivity Testing on Offensive Progressions [6.19084217044276]
  Mitigating explicit and implicit biases in Large Language Models (LLMs) has become a critical focus in the field of natural language processing. We introduce the Sensitivity Testing on Offensive Progressions dataset, which includes 450 offensive progressions containing 2,700 unique sentences. Our findings reveal that even the best-performing models detect bias inconsistently, with success rates ranging from 19.3% to 69.8%.
  arXiv  Detail & Related papers  (2024-09-20T18:34:38Z)
• Self-supervised co-salient object detection via feature correspondence at multiple scales [27.664016341526988]
  This paper introduces a novel two-stage self-supervised approach for detecting co-occurring salient objects (CoSOD) in image groups without requiring segmentation annotations. We train a self-supervised network that detects co-salient regions by computing local patch-level feature correspondences across images. In experiments on three CoSOD benchmark datasets, our model outperforms the corresponding state-of-the-art models by a huge margin.
  arXiv  Detail & Related papers  (2024-03-17T06:21:21Z)
• Benchmarking Zero-Shot Robustness of Multimodal Foundation Models: A Pilot Study [61.65123150513683]
  multimodal foundation models, such as CLIP, produce state-of-the-art zero-shot results. It is reported that these models close the robustness gap by matching the performance of supervised models trained on ImageNet. We show that CLIP leads to a significant robustness drop compared to supervised ImageNet models on our benchmark.
  arXiv  Detail & Related papers  (2024-03-15T17:33:49Z)
• Unsupervised and semi-supervised co-salient object detection via segmentation frequency statistics [24.397029434125873]
  We develop an unsupervised method to detect co-occurring salient objects in an image group. For the first time, we show that a large unlabeled dataset e.g. ImageNet-1k can be effectively leveraged to improve unsupervised CoSOD performance. To avoid propagating erroneous signals from predictions on unlabeled data, we propose a confidence estimation module.
  arXiv  Detail & Related papers  (2023-11-11T19:47:16Z)
• GEO-Bench: Toward Foundation Models for Earth Monitoring [139.77907168809085]
  We propose a benchmark comprised of six classification and six segmentation tasks. This benchmark will be a driver of progress across a variety of Earth monitoring tasks.
  arXiv  Detail & Related papers  (2023-06-06T16:16:05Z)
• Zero-shot Video Moment Retrieval With Off-the-Shelf Models [17.254884920876695]
  We propose a zero-shot simple approach for one such task, Video Moment Retrieval (VMR) We vastly improve performance of previous zero-shot approaches by at least 2.5x on all metrics. We also show that our zero-shot approach beats non-pretrained supervised models on the Recall metrics and comes very close on mAP metrics.
  arXiv  Detail & Related papers  (2022-11-03T23:11:04Z)
• COLO: A Contrastive Learning based Re-ranking Framework for One-Stage Summarization [84.70895015194188]
  We propose a Contrastive Learning based re-ranking framework for one-stage summarization called COLO. COLO boosts the extractive and abstractive results of one-stage systems on CNN/DailyMail benchmark to 44.58 and 46.33 ROUGE-1 score.
  arXiv  Detail & Related papers  (2022-09-29T06:11:21Z)
• Improving Visual Grounding by Encouraging Consistent Gradient-based Explanations [58.442103936918805]
  We show that Attention Mask Consistency produces superior visual grounding results than previous methods. AMC is effective, easy to implement, and is general as it can be adopted by any vision-language model.
  arXiv  Detail & Related papers  (2022-06-30T17:55:12Z)
• BRIO: Bringing Order to Abstractive Summarization [107.97378285293507]
  We propose a novel training paradigm which assumes a non-deterministic distribution. Our method achieves a new state-of-the-art result on the CNN/DailyMail (47.78 ROUGE-1) and XSum (49.07 ROUGE-1) datasets.
  arXiv  Detail & Related papers  (2022-03-31T05:19:38Z)
• Adversarial Filters of Dataset Biases [96.090959788952]
  Large neural models have demonstrated human-level performance on language and vision benchmarks. Their performance degrades considerably on adversarial or out-of-distribution samples. We propose AFLite, which adversarially filters such dataset biases.
  arXiv  Detail & Related papers  (2020-02-10T21:59:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.