Using Multimodal and Language-Agnostic Sentence Embeddings for Abstractive Summarization

• URL: http://arxiv.org/abs/2603.08282v1
• Date: Mon, 09 Mar 2026 11:55:32 GMT
• Title: Using Multimodal and Language-Agnostic Sentence Embeddings for Abstractive Summarization
• Abstract summary: Abstractive summarization aims to generate concise summaries by creating new sentences, allowing for flexible rephrasing.<n>This approach can be vulnerable to inaccuracies, particularly hallucinations' where the model introduces non-existent information.<n>We leverage the use of multimodal and multilingual sentence embeddings derived from pretrained models such as LaBSE, SONAR, and BGE-M3, and feed them into a modified BART-based French model.<n>A Named Entity Injection mechanism that appends tokenized named entities to the decoder input is introduced, in order to improve the factual consistency of the generated summary.
• Score: 10.892363692521222
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Abstractive summarization aims to generate concise summaries by creating new sentences, allowing for flexible rephrasing. However, this approach can be vulnerable to inaccuracies, particularly `hallucinations' where the model introduces non-existent information. In this paper, we leverage the use of multimodal and multilingual sentence embeddings derived from pretrained models such as LaBSE, SONAR, and BGE-M3, and feed them into a modified BART-based French model. A Named Entity Injection mechanism that appends tokenized named entities to the decoder input is introduced, in order to improve the factual consistency of the generated summary. Our novel framework, SBARThez, is applicable to both text and speech inputs and supports cross-lingual summarization; it shows competitive performance relative to token-level baselines, especially for low-resource languages, while generating more concise and abstract summaries.

Related papers

• Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Models [60.00178316095646]
  Sentence embedding is essential for many NLP tasks, with contrastive learning methods achieving strong performance using datasets like NLI.<n>Recent studies leverage large language models (LLMs) to generate sentence pairs, reducing annotation dependency.<n>We propose a method for controlling the generation direction of LLMs in the latent space. Unlike unconstrained generation, the controlled approach ensures meaningful semantic divergence.<n> Experiments on multiple benchmarks demonstrate that our method achieves new SOTA performance with a modest cost in ranking sentence synthesis.
  arXiv  Detail & Related papers  (2025-02-19T12:07:53Z)
• Boosting the Capabilities of Compact Models in Low-Data Contexts with Large Language Models and Retrieval-Augmented Generation [2.9921619703037274]
  We propose a retrieval augmented generation (RAG) framework backed by a large language model (LLM) to correct the output of a smaller model for the linguistic task of morphological glossing. We leverage linguistic information to make up for the lack of data and trainable parameters, while allowing for inputs from written descriptive grammars interpreted and distilled through an LLM. We show that a compact, RAG-supported model is highly effective in data-scarce settings, achieving a new state-of-the-art for this task and our target languages.
  arXiv  Detail & Related papers  (2024-10-01T04:20:14Z)
• Language Representations Can be What Recommenders Need: Findings and Potentials [57.90679739598295]
  We show that item representations, when linearly mapped from advanced LM representations, yield superior recommendation performance.<n>This outcome suggests the possible homomorphism between the advanced language representation space and an effective item representation space for recommendation.<n>Our findings highlight the connection between language modeling and behavior modeling, which can inspire both natural language processing and recommender system communities.
  arXiv  Detail & Related papers  (2024-07-07T17:05:24Z)
• Towards Enhancing Coherence in Extractive Summarization: Dataset and Experiments with LLMs [70.15262704746378]
  We propose a systematically created human-annotated dataset consisting of coherent summaries for five publicly available datasets and natural language user feedback. Preliminary experiments with Falcon-40B and Llama-2-13B show significant performance improvements (10% Rouge-L) in terms of producing coherent summaries.
  arXiv  Detail & Related papers  (2024-07-05T20:25:04Z)
• An Extractive-and-Abstractive Framework for Source Code Summarization [28.553366270065656]
  Code summarization aims to automatically generate summaries/comments for a given code snippet in the form of natural language. We propose a novel extractive-and-abstractive framework to generate human-written-like summaries with preserved factual details.
  arXiv  Detail & Related papers  (2022-06-15T02:14:24Z)
• Dialogue Summarization with Supporting Utterance Flow Modeling and Fact Regularization [58.965859508695225]
  We propose an end-to-end neural model for dialogue summarization with two novel modules. The supporting utterance flow modeling helps to generate a coherent summary by smoothly shifting the focus from the former utterances to the later ones. The fact regularization encourages the generated summary to be factually consistent with the ground-truth summary during model training.
  arXiv  Detail & Related papers  (2021-08-03T03:09:25Z)
• Semantic Extractor-Paraphraser based Abstractive Summarization [40.05739160204135]
  We propose an extractor-paraphraser based abstractive summarization system that exploits semantic overlap. Our model outperforms the state-of-the-art baselines in terms of ROUGE, METEOR and word similarity (WMS)
  arXiv  Detail & Related papers  (2021-05-04T05:24:28Z)
• SLM: Learning a Discourse Language Representation with Sentence Unshuffling [53.42814722621715]
  We introduce Sentence-level Language Modeling, a new pre-training objective for learning a discourse language representation. We show that this feature of our model improves the performance of the original BERT by large margins.
  arXiv  Detail & Related papers  (2020-10-30T13:33:41Z)
• Multi-Fact Correction in Abstractive Text Summarization [98.27031108197944]
  Span-Fact is a suite of two factual correction models that leverages knowledge learned from question answering models to make corrections in system-generated summaries via span selection. Our models employ single or multi-masking strategies to either iteratively or auto-regressively replace entities in order to ensure semantic consistency w.r.t. the source text. Experiments show that our models significantly boost the factual consistency of system-generated summaries without sacrificing summary quality in terms of both automatic metrics and human evaluation.
  arXiv  Detail & Related papers  (2020-10-06T02:51:02Z)
• An Effective Contextual Language Modeling Framework for Speech Summarization with Augmented Features [13.97006782398121]
  Bidirectional Representations from Transformers (BERT) model was proposed and has achieved record-breaking success on many natural language processing tasks. We explore the incorporation of confidence scores into sentence representations to see if such an attempt could help alleviate the negative effects caused by imperfect automatic speech recognition. We validate the effectiveness of our proposed method on a benchmark dataset.
  arXiv  Detail & Related papers  (2020-06-01T18:27:48Z)
• Exploring Explainable Selection to Control Abstractive Summarization [51.74889133688111]
  We develop a novel framework that focuses on explainability. A novel pair-wise matrix captures the sentence interactions, centrality, and attribute scores. A sentence-deployed attention mechanism in the abstractor ensures the final summary emphasizes the desired content.
  arXiv  Detail & Related papers  (2020-04-24T14:39:34Z)
• StructSum: Summarization via Structured Representations [27.890477913486787]
  Abstractive text summarization aims at compressing the information of a long source document into a condensed summary. Despite advances in modeling techniques, abstractive summarization models still suffer from several key challenges. We propose a framework based on document-level structure induction for summarization to address these challenges.
  arXiv  Detail & Related papers  (2020-03-01T20:32:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.