Adaptation of Embedding Models to Financial Filings via LLM Distillation

• URL: http://arxiv.org/abs/2512.08088v1
• Date: Mon, 08 Dec 2025 22:43:14 GMT
• Title: Adaptation of Embedding Models to Financial Filings via LLM Distillation
• Authors: Eliot Brenner, Dominic Seyler, Manjunath Hegde, Andrei Simion, Koustuv Dasgupta, Bing Xiang,
• Abstract summary: This paper introduces a scalable pipeline that trains specialized models from an unlabeled corpus using a general purpose retrieval embedding model as foundation.<n>Our method yields an average of 27.7% improvement in MRR$textt@$5, 44.6% improvement in mean DCG$textt@$5 across 14 financial filing types measured over 21,800 query-document pairs.
• Score: 10.744318713371383
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite advances in generative large language models (LLMs), practical application of specialized conversational AI agents remains constrained by computation costs, latency requirements, and the need for precise domain-specific relevance measures. While existing embedding models address the first two constraints, they underperform on information retrieval in specialized domains like finance. This paper introduces a scalable pipeline that trains specialized models from an unlabeled corpus using a general purpose retrieval embedding model as foundation. Our method yields an average of 27.7% improvement in MRR$\texttt{@}$5, 44.6% improvement in mean DCG$\texttt{@}$5 across 14 financial filing types measured over 21,800 query-document pairs, and improved NDCG on 3 of 4 document classes in FinanceBench. We adapt retrieval embeddings (bi-encoder) for RAG, not LLM generators, using LLM-judged relevance to distill domain knowledge into a compact retriever. There are prior works which pair synthetically generated queries with real passages to directly fine-tune the retrieval model. Our pipeline differs from these by introducing interaction between student and teacher models that interleaves retrieval-based mining of hard positive/negative examples from the unlabeled corpus with iterative retraining of the student model's weights using these examples. Each retrieval iteration uses the refined student model to mine the corpus for progressively harder training examples for the subsequent training iteration. The methodology provides a cost-effective solution to bridging the gap between general-purpose models and specialized domains without requiring labor-intensive human annotation.

Related papers

• HuggingR$^{4}$: A Progressive Reasoning Framework for Discovering Optimal Model Companions [50.61510609116118]
  HuggingR$4$ is a novel framework that combines Reasoning, Retrieval, Refinement, and Reflection to efficiently select models.<n>It attains a workability rate of 92.03% and a reasonability rate of 82.46%, surpassing existing method by 26.51% and 33.25% respectively.
  arXiv  Detail & Related papers  (2025-11-24T03:13:45Z)
• MARAG-R1: Beyond Single Retriever via Reinforcement-Learned Multi-Tool Agentic Retrieval [50.30107119622642]
  Large Language Models (LLMs) excel at reasoning and generation but are inherently limited by static pretraining data.<n>Retrieval-Augmented Generation (RAG) addresses this issue by grounding LLMs in external knowledge.<n>MarAG-R1 is a reinforcement-learned multi-tool RAG framework that enables LLMs to dynamically coordinate multiple retrieval mechanisms.
  arXiv  Detail & Related papers  (2025-10-31T15:51:39Z)
• Large Reasoning Embedding Models: Towards Next-Generation Dense Retrieval Paradigm [16.78399933831573]
  We propose the Large Reasoning Embedding Model (LREM), which integrates reasoning processes into representation learning.<n>For difficult queries, LREM first conducts reasoning to achieve a deep understanding of the original query, and then produces a reasoning-augmented query embedding for retrieval.<n>This reasoning process effectively bridges the semantic gap between original queries and target items, significantly improving retrieval accuracy.
  arXiv  Detail & Related papers  (2025-10-16T05:37:39Z)
• CustomIR: Unsupervised Fine-Tuning of Dense Embeddings for Known Document Corpora [0.0]
  CustomIR is a framework for unsupervised adaptation of language embedding models to domain-specific corpora.<n>Our experiments show that CustomIR consistently improves retrieval effectiveness with small models gaining up to 2.3 points in Recall@10.<n>These results highlight that targeted synthetic fine-tuning offers a scalable and cost-efficient strategy for increasing domain-specific performance.
  arXiv  Detail & Related papers  (2025-09-30T00:25:47Z)
• Don't Retrieve, Generate: Prompting LLMs for Synthetic Training Data in Dense Retrieval [0.0]
  Training effective dense retrieval models typically relies on hard negative (HN) examples mined from large document corpora.<n>We propose an end-to-end pipeline where a Large Language Model (LLM) first generates a query from a passage and then produces a hard negative example using only the generated query text.<n>Our dataset comprises 7,250 arXiv abstracts spanning diverse domains including mathematics, physics, computer science, and related fields.
  arXiv  Detail & Related papers  (2025-04-20T08:34:10Z)
• FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Models [35.40065954148091]
  FINE is a method based on the Learngene framework to initializing downstream networks leveraging pre-trained models. It decomposes pre-trained knowledge into the product of matrices (i.e., $U$, $Sigma$, and $V$), where $U$ and $V$ are shared across network blocks as learngenes'' It consistently outperforms direct pre-training, particularly for smaller models, achieving state-of-the-art results across variable model sizes.
  arXiv  Detail & Related papers  (2024-09-28T08:57:17Z)
• Few-shot Prompting for Pairwise Ranking: An Effective Non-Parametric Retrieval Model [18.111868378615206]
  We propose a pairwise few-shot ranker that achieves a close performance to that of a supervised model without requiring any complex training pipeline. Our method also achieves a close performance to that of a supervised model without requiring any complex training pipeline.
  arXiv  Detail & Related papers  (2024-09-26T11:19:09Z)
• List-aware Reranking-Truncation Joint Model for Search and Retrieval-augmented Generation [80.12531449946655]
  We propose a Reranking-Truncation joint model (GenRT) that can perform the two tasks concurrently. GenRT integrates reranking and truncation via generative paradigm based on encoder-decoder architecture. Our method achieves SOTA performance on both reranking and truncation tasks for web search and retrieval-augmented LLMs.
  arXiv  Detail & Related papers  (2024-02-05T06:52:53Z)
• You can't pick your neighbors, or can you? When and how to rely on retrieval in the $k$NN-LM [65.74934004876914]
  Retrieval-enhanced language models (LMs) condition their predictions on text retrieved from large external datastores. One such approach, the $k$NN-LM, interpolates any existing LM's predictions with the output of a $k$-nearest neighbors model. We empirically measure the effectiveness of our approach on two English language modeling datasets.
  arXiv  Detail & Related papers  (2022-10-28T02:57:40Z)
• The Right Tool for the Job: Matching Model and Instance Complexities [62.95183777679024]
  As NLP models become larger, executing a trained model requires significant computational resources incurring monetary and environmental costs. We propose a modification to contextual representation fine-tuning which, during inference, allows for an early (and fast) "exit" We test our proposed modification on five different datasets in two tasks: three text classification datasets and two natural language inference benchmarks.
  arXiv  Detail & Related papers  (2020-04-16T04:28:08Z)
• Pre-training Tasks for Embedding-based Large-scale Retrieval [68.01167604281578]
  We consider the large-scale query-document retrieval problem. Given a query (e.g., a question), return the set of relevant documents from a large document corpus. We show that the key ingredient of learning a strong embedding-based Transformer model is the set of pre-training tasks.
  arXiv  Detail & Related papers  (2020-02-10T16:44:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.