FLeW: Facet-Level and Adaptive Weighted Representation Learning of Scientific Documents

• URL: http://arxiv.org/abs/2509.07531v1
• Date: Tue, 09 Sep 2025 09:08:44 GMT
• Title: FLeW: Facet-Level and Adaptive Weighted Representation Learning of Scientific Documents
• Authors: Zheng Dou, Deqing Wang, Fuzhen Zhuang, Jian Ren, Yanlin Hu,
• Abstract summary: We propose a new method that unifies the three approaches for better representations, namely FLeW.<n> FLeW integrates three facet-level embeddings into a task-specific document embedding without task-aware fine-tuning.<n>Experiments show the applicability and robustness of FLeW across multiple scientific tasks and fields, compared to prior models.
• Score: 25.82731569190015
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scientific document representation learning provides powerful embeddings for various tasks, while current methods face challenges across three approaches. 1) Contrastive training with citation-structural signals underutilizes citation information and still generates single-vector representations. 2) Fine-grained representation learning, which generates multiple vectors at the sentence or aspect level, requires costly integration and lacks domain generalization. 3) Task-aware learning depends on manually predefined task categorization, overlooking nuanced task distinctions and requiring extra training data for task-specific modules. To address these problems, we propose a new method that unifies the three approaches for better representations, namely FLeW. Specifically, we introduce a novel triplet sampling method that leverages citation intent and frequency to enhance citation-structural signals for training. Citation intents (background, method, result), aligned with the general structure of scientific writing, facilitate a domain-generalized facet partition for fine-grained representation learning. Then, we adopt a simple weight search to adaptively integrate three facet-level embeddings into a task-specific document embedding without task-aware fine-tuning. Experiments show the applicability and robustness of FLeW across multiple scientific tasks and fields, compared to prior models.

Related papers

• Task-Adapter++: Task-specific Adaptation with Order-aware Alignment for Few-shot Action Recognition [33.22316608406554]
  We propose a parameter-efficient dual adaptation method for both image and text encoders.<n>Specifically, we design a task-specific adaptation for the image encoder so that the most discriminative information can be well noticed during feature extraction.<n>We develop an innovative fine-grained cross-modal alignment strategy that actively maps visual features to reside in the same temporal stage as semantic descriptions.
  arXiv  Detail & Related papers  (2025-05-09T12:34:10Z)
• Generative Compositor for Few-Shot Visual Information Extraction [60.663887314625164]
  We propose a novel generative model, named Generative generative spatialtor, to address the challenge of few-shot VIE.<n>Generative generative spatialtor is a hybrid pointer-generator network that emulates the operations of a compositor by retrieving words from the source text.<n>The proposed method achieves highly competitive results in the full-sample training, while notably outperforms the baseline in the 1-shot, 5-shot, and 10-shot settings.
  arXiv  Detail & Related papers  (2025-03-21T04:56:24Z)
• Learning Task Representations from In-Context Learning [73.72066284711462]
  Large language models (LLMs) have demonstrated remarkable proficiency in in-context learning.<n>We introduce an automated formulation for encoding task information in ICL prompts as a function of attention heads.<n>We show that our method's effectiveness stems from aligning the distribution of the last hidden state with that of an optimally performing in-context-learned model.
  arXiv  Detail & Related papers  (2025-02-08T00:16:44Z)
• Boosting Short Text Classification with Multi-Source Information Exploration and Dual-Level Contrastive Learning [12.377363857246602]
  We propose a novel model named MI-DELIGHT for short text classification.<n>It first performs multi-source information exploration to alleviate the sparsity issues.<n>Then, the graph learning approach is adopted to learn the representation of short texts.
  arXiv  Detail & Related papers  (2025-01-16T00:26:15Z)
• Probing Representations for Document-level Event Extraction [30.523959637364484]
  This work is the first to apply the probing paradigm to representations learned for document-level information extraction. We designed eight embedding probes to analyze surface, semantic, and event-understanding capabilities relevant to document-level event extraction. We found that trained encoders from these models yield embeddings that can modestly improve argument detections and labeling but only slightly enhance event-level tasks.
  arXiv  Detail & Related papers  (2023-10-23T19:33:04Z)
• Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding [52.723297744257536]
  Pre-trained language models (LMs) have shown effectiveness in scientific literature understanding tasks. We propose a multi-task contrastive learning framework, SciMult, to facilitate common knowledge sharing across different literature understanding tasks.
  arXiv  Detail & Related papers  (2023-05-23T16:47:22Z)
• SciRepEval: A Multi-Format Benchmark for Scientific Document Representations [52.01865318382197]
  We introduce SciRepEval, the first comprehensive benchmark for training and evaluating scientific document representations. We show how state-of-the-art models like SPECTER and SciNCL struggle to generalize across the task formats. A new approach that learns multiple embeddings per document, each tailored to a different format, can improve performance.
  arXiv  Detail & Related papers  (2022-11-23T21:25:39Z)
• Fast Inference and Transfer of Compositional Task Structures for Few-shot Task Generalization [101.72755769194677]
  We formulate it as a few-shot reinforcement learning problem where a task is characterized by a subtask graph. Our multi-task subtask graph inferencer (MTSGI) first infers the common high-level task structure in terms of the subtask graph from the training tasks. Our experiment results on 2D grid-world and complex web navigation domains show that the proposed method can learn and leverage the common underlying structure of the tasks for faster adaptation to the unseen tasks.
  arXiv  Detail & Related papers  (2022-05-25T10:44:25Z)
• Improving Multi-task Generalization Ability for Neural Text Matching via Prompt Learning [54.66399120084227]
  Recent state-of-the-art neural text matching models (PLMs) are hard to generalize to different tasks. We adopt a specialization-generalization training strategy and refer to it as Match-Prompt. In specialization stage, descriptions of different matching tasks are mapped to only a few prompt tokens. In generalization stage, text matching model explores the essential matching signals by being trained on diverse multiple matching tasks.
  arXiv  Detail & Related papers  (2022-04-06T11:01:08Z)
• Grad2Task: Improved Few-shot Text Classification Using Gradients for Task Representation [24.488427641442694]
  We propose a novel conditional neural process-based approach for few-shot text classification. Our key idea is to represent each task using gradient information from a base model. Our approach outperforms traditional fine-tuning, sequential transfer learning, and state-of-the-art meta learning approaches.
  arXiv  Detail & Related papers  (2022-01-27T15:29:30Z)
• Curriculum-Meta Learning for Order-Robust Continual Relation Extraction [12.494209368988253]
  We propose a novel curriculum-meta learning method to tackle the challenges of continual relation extraction. We combine meta learning and curriculum learning to quickly adapt model parameters to a new task. We present novel difficulty-based metrics to quantitatively measure the extent of order-sensitivity of a given model.
  arXiv  Detail & Related papers  (2021-01-06T08:52:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.