Learn Before Represent: Bridging Generative and Contrastive Learning for Domain-Specific LLM Embeddings

• URL: http://arxiv.org/abs/2601.11124v1
• Date: Fri, 16 Jan 2026 09:35:29 GMT
• Title: Learn Before Represent: Bridging Generative and Contrastive Learning for Domain-Specific LLM Embeddings
• Authors: Xiaoyu Liang, Yuchen Peng, Jiale Luo, Wenhao Wang, Haoji Hu, Xincheng Zhou,
• Abstract summary: Large Language Models (LLMs) adapted via contrastive learning excel in general representation learning but struggle in vertical domains like chemistry and law.<n>This work identifies a core bottleneck: the prevailing LLM+CL'' paradigm focuses on semantic alignment but cannot perform knowledge acquisition.<n>We propose Learn Before Represent, a novel two-stage framework for building accurate and robust representations in vertical domains.
• Score: 14.859728745469354
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) adapted via contrastive learning excel in general representation learning but struggle in vertical domains like chemistry and law, primarily due to a lack of domain-specific knowledge. This work identifies a core bottleneck: the prevailing ``LLM+CL'' paradigm focuses on semantic alignment but cannot perform knowledge acquisition, leading to failures on specialized terminology. To bridge this gap, we propose Learn Before Represent (LBR), a novel two-stage framework. LBR first injects domain knowledge via an Information Bottleneck-Constrained Generative Learning stage, preserving the LLM's causal attention to maximize knowledge acquisition while compressing semantics. It then performs Generative-Refined Contrastive Learning on the compressed representations for alignment. This approach maintains architectural consistency and resolves the objective conflict between generative and contrastive learning. Extensive experiments on medical, chemistry, and code retrieval tasks show that LBR significantly outperforms strong baselines. Our work establishes a new paradigm for building accurate and robust representations in vertical domains.

Related papers

• KUDA: Knowledge Unlearning by Deviating Representation for Large Language Models [26.418820118903852]
  Large language models (LLMs) acquire a large amount of knowledge through pre-training on vast and diverse corpora.<n>LLMs unlearning is a promising technique to reduce risks associated with sensitive, copyrighted, or harmful content in training data.<n>We propose Knowledge Unlearning by Deviating representAtion (KUDA) to achieve effective unlearning at the knowledge level of LLMs.
  arXiv  Detail & Related papers  (2026-02-22T17:16:49Z)
• Language as an Anchor: Preserving Relative Visual Geometry for Domain Incremental Learning [8.952803050083203]
  Key challenge in Domain Incremental Learning is to continually learn under shifting distributions.<n>We propose LAVA, a novel DIL framework that replaces direct feature alignment with relative alignment driven by a text-based reference anchor.<n> experiments on standard DIL benchmarks demonstrate that LAVA achieves significant performance improvements over state-of-the-arts.
  arXiv  Detail & Related papers  (2025-11-18T12:06:55Z)
• GRIL: Knowledge Graph Retrieval-Integrated Learning with Large Language Models [59.72897499248909]
  We propose a novel graph retriever trained end-to-end with Large Language Models (LLMs)<n>Within the extracted subgraph, structural knowledge and semantic features are encoded via soft tokens and the verbalized graph, respectively, which are infused into the LLM together.<n>Our approach consistently achieves state-of-the-art performance, validating the strength of joint graph-LLM optimization for complex reasoning tasks.
  arXiv  Detail & Related papers  (2025-09-20T02:38:00Z)
• Are Unified Vision-Language Models Necessary: Generalization Across Understanding and Generation [50.22361866757033]
  unified vision-language models (VLMs) integrate both visual understanding and generation capabilities.<n>This paper systematically investigates the generalization across understanding and generation tasks in unifiedVLMs.
  arXiv  Detail & Related papers  (2025-05-29T03:40:21Z)
• OntoTune: Ontology-Driven Self-training for Aligning Large Language Models [36.707858872631945]
  Training on large-scale corpora often fails to effectively organize domain knowledge of Large Language Models.<n>Inspired by how humans connect concepts and organize knowledge through mind maps, we propose an ontology-driven self-training framework called OntoTune.<n>We conduct our study in the medical domain to evaluate the effectiveness of OntoTune.
  arXiv  Detail & Related papers  (2025-02-08T07:38:45Z)
• BLADE: Enhancing Black-box Large Language Models with Small Domain-Specific Models [56.89958793648104]
  Large Language Models (LLMs) are versatile and capable of addressing a diverse range of tasks. Previous approaches either conduct continuous pre-training with domain-specific data or employ retrieval augmentation to support general LLMs. We present a novel framework named BLADE, which enhances Black-box LArge language models with small Domain-spEcific models.
  arXiv  Detail & Related papers  (2024-03-27T08:57:21Z)
• The Strong Pull of Prior Knowledge in Large Language Models and Its Impact on Emotion Recognition [74.04775677110179]
  In-context Learning (ICL) has emerged as a powerful paradigm for performing natural language tasks with Large Language Models (LLM) We show that LLMs have strong yet inconsistent priors in emotion recognition that ossify their predictions. Our results suggest that caution is needed when using ICL with larger LLMs for affect-centered tasks outside their pre-training domain.
  arXiv  Detail & Related papers  (2024-03-25T19:07:32Z)
• Fusing Domain-Specific Content from Large Language Models into Knowledge Graphs for Enhanced Zero Shot Object State Classification [1.1161827123148225]
  This study investigates the potential of Large Language Models (LLMs) in generating and providing domain-specific information.<n>To achieve this, an LLM is integrated into a pipeline that utilizes Knowledge Graphs and pre-trained semantic vectors.<n>Our findings reveal that the integration of LLM-based embeddings, in combination with general-purpose pre-trained embeddings, leads to substantial performance improvements.
  arXiv  Detail & Related papers  (2024-03-18T18:08:44Z)
• Knowledge Plugins: Enhancing Large Language Models for Domain-Specific Recommendations [50.81844184210381]
  We propose a general paradigm that augments large language models with DOmain-specific KnowledgE to enhance their performance on practical applications, namely DOKE. This paradigm relies on a domain knowledge extractor, working in three steps: 1) preparing effective knowledge for the task; 2) selecting the knowledge for each specific sample; and 3) expressing the knowledge in an LLM-understandable way.
  arXiv  Detail & Related papers  (2023-11-16T07:09:38Z)
• Integrating Language Guidance into Vision-based Deep Metric Learning [78.18860829585182]
  We propose to learn metric spaces which encode semantic similarities as embedding space. These spaces should be transferable to classes beyond those seen during training. This causes learned embedding spaces to encode incomplete semantic context and misrepresent the semantic relation between classes.
  arXiv  Detail & Related papers  (2022-03-16T11:06:50Z)
• Domain-oriented Language Pre-training with Adaptive Hybrid Masking and Optimal Transport Alignment [43.874781718934486]
  We provide a general domain-oriented approach to adapt pre-trained language models for different application domains. To preserve phrase knowledge effectively, we build a domain phrase pool as auxiliary training tool. We introduce Cross Entity Alignment to leverage entity association as weak supervision to augment the semantic learning of pre-trained models.
  arXiv  Detail & Related papers  (2021-12-01T15:47:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.