Attention Expansion: Enhancing Keyphrase Extraction from Long Documents with Attention-Augmented Contextualized Embeddings

• URL: http://arxiv.org/abs/2606.10716v2
• Date: Thu, 11 Jun 2026 13:08:37 GMT
• Title: Attention Expansion: Enhancing Keyphrase Extraction from Long Documents with Attention-Augmented Contextualized Embeddings
• Abstract summary: We propose an attention expansion mechanism that augments PLM token representations with information from surrounding out-of-context chunks.<n>We evaluate our approach across five PLM backbones, including general-purpose, scientific, task-specific, and long-context encoders.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pre-trained language models (PLMs) have achieved strong performance in keyphrase extraction (KPE), largely due to their ability to generate rich contextualized representations. However, long-document KPE remains challenging because salient keyphrase evidence may be scattered across distant document sections that cannot be jointly captured within the limited context window of most PLMs. Although long-context large language models (LLMs) can process broader textual contexts, their computational cost limits their practicality for efficient and high-throughput KPE. To overcome this limitation, we propose an attention expansion mechanism that augments PLM token representations with information from surrounding out-of-context chunks using pre-trained word embeddings. The proposed mechanism expands the effective contextual scope of PLM-based KPE models without requiring full-document attention or expensive LLM-based inference. We evaluate our approach across five PLM backbones, including general-purpose, scientific, task-specific, and long-context encoders, using two training regimes and five benchmark corpora from scientific and news domains. Experimental results demonstrate that attention expansion consistently enhances KPE performance across all evaluation settings, outperforming state-of-the-art models and yielding notable improvements in F1 score. The improvements extend to domain-specific, task-specialized, and native long-context models, showing that the proposed mechanism provides complementary information rather than merely compensating for limited input length. These results establish attention expansion as an efficient and effective strategy for long-document KPE.

Related papers

• DMRL: Document-Mediated Reinforcement Learning for Skill Optimization in Advertising Recommendation [18.143707250252927]
  Document-Mediated Reinforcement Learning (DMRL) is a skill self-evolution framework that models skill document optimization as a sequence of structured editing actions.<n>DMRL was deployed on a large-scale short-video ads platform and extensive empirical evaluation shows that it outperforms state-of-the-art baselines across key advertising metrics.
  arXiv  Detail & Related papers  (2026-09-02T06:32:48Z)
• Up to 36x Speedup: Mask-based Parallel Inference Paradigm for Key Information Extraction in MLLMs [22.76757502541604]
  We introduce PIP: a Parallel Inference Paradigm for Key Information Extraction (KIE)<n>Our approach reformulates the problem by using "[mask]" tokens as placeholders for all target values, enabling their simultaneous generation in a single forward pass.<n> Experimental results show that our PIP-models achieve a 5-36x inference speedup with negligible performance degradation compared to traditional autoregressive base models.
  arXiv  Detail & Related papers  (2026-01-27T13:45:30Z)
• Structured Attention Matters to Multimodal LLMs in Document Understanding [52.37530640460363]
  We investigate how input format influences document comprehension performance.<n>We discover that raw OCR text often impairs rather than improves MLLMs' performance.<n>We propose a novel structure-preserving approach that encodes document elements using the LaTex paradigm.
  arXiv  Detail & Related papers  (2025-06-19T07:16:18Z)
• CSE-SFP: Enabling Unsupervised Sentence Representation Learning via a Single Forward Pass [3.0566617373924325]
  Recent advances in pre-trained language models (PLMs) have driven remarkable progress in this field.<n>We propose CSE-SFP, an innovative method that exploits the structural characteristics of generative models.<n>We show that CSE-SFP not only produces higher-quality embeddings but also significantly reduces both training time and memory consumption.
  arXiv  Detail & Related papers  (2025-05-01T08:27:14Z)
• Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation [60.22622442950905]
  Large Language Models (LLMs) often struggle to process and generate coherent context when the number of input tokens exceeds the pre-trained length.<n>Recent advancements in long-context extension have significantly expanded the context window of LLMs.<n>We propose Dimension-Wise Positional Embeddings Manipulation (DPE) to extrapolate the context window of LLMs.
  arXiv  Detail & Related papers  (2025-04-26T08:46:10Z)
• PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention [73.26995918610669]
  Large Language Models (LLMs) face efficiency bottlenecks due to the quadratic complexity of the attention mechanism when processing long contexts.<n>We introduce PowerAttention, a novel sparse attention design that facilitates effective and complete context extension.<n>Experiments demonstrate that PowerAttention outperforms existing static sparse attention methods by $5sim 40%$.
  arXiv  Detail & Related papers  (2025-03-05T15:24:11Z)
• Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference [6.699062502635993]
  Long-context inputs are crucial for the effective utilization of large language models (LLMs)<n>We propose an efficient, training-free prompt compression method that retains key information within compressed prompts.<n>We develop Evaluator Head-based Prompt Compression, which enables LLMs to rapidly "skim through" input prompts.
  arXiv  Detail & Related papers  (2025-01-22T15:33:17Z)
• StructFormer: Document Structure-based Masked Attention and its Impact on Language Model Pre-Training [20.79815837785261]
  This study focuses on empirically assessing the influence of global attention on BERT pre-training. We create an extensive corpus of structure-aware text through arXiv data, alongside a text-only counterpart. Our analysis underscores the significance of incorporating document structure into LM models, demonstrating their capacity to excel in more abstract tasks.
  arXiv  Detail & Related papers  (2024-11-25T17:57:52Z)
• SEGMENT+: Long Text Processing with Short-Context Language Models [53.40059130780192]
  SEGMENT+ is a framework that enables LMs to handle extended inputs within limited context windows efficiently. SEGMENT+ utilizes structured notes and a filtering module to manage information flow, resulting in a system that is both controllable and interpretable.
  arXiv  Detail & Related papers  (2024-10-09T03:40:22Z)
• CELA: Cost-Efficient Language Model Alignment for CTR Prediction [70.65910069412944]
  Click-Through Rate (CTR) prediction holds a paramount position in recommender systems.<n>Recent efforts have sought to mitigate these challenges by integrating Pre-trained Language Models (PLMs)<n>We propose textbfCost-textbfEfficient textbfLanguage Model textbfAlignment (textbfCELA) for CTR prediction.
  arXiv  Detail & Related papers  (2024-05-17T07:43:25Z)
• Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study [76.52997424694767]
  We present an in-depth empirical study of keyphrase extraction and keyphrase generation using pre-trained language models. We show that PLMs have competitive high-resource performance and state-of-the-art low-resource performance. Further results show that in-domain BERT-like PLMs can be used to build strong and data-efficient keyphrase generation models.
  arXiv  Detail & Related papers  (2022-12-20T13:20:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.