Zero-Shot Contextual Embeddings via Offline Synthetic Corpus Generation

• URL: http://arxiv.org/abs/2506.23662v1
• Date: Mon, 30 Jun 2025 09:38:50 GMT
• Title: Zero-Shot Contextual Embeddings via Offline Synthetic Corpus Generation
• Abstract summary: ZEST is a zero-shot contextual adaptation framework.<n>It replaces real corpus access with a one-time offline synthesis of a compact proxy.
• Score: 4.313454680394974
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Context-aware embedding methods boost retrieval accuracy by conditioning on corpus statistics (e.g., term co-occurrence and topical patterns) extracted from neighboring documents. However, this context-aware approach requires access to the target corpus or requires domain-specific finetuning, posing practical barriers in privacy-sensitive or resource-constrained settings. We present ZEST, a zero-shot contextual adaptation framework that replaces real corpus access with a one-time offline synthesis of a compact proxy. Given only a handful exemplar documents representative of the general target domain, we use a multi-step hierarchical procedure to generate a synthetic context corpus of several hundred documents that aims to emulate key domain-specific distributions. At inference, the frozen context-aware encoder uses this proxy corpus -- without any finetuning or target corpus access -- to produce domain-adapted embeddings. Across the MTEB benchmark, ZEST's zero-shot synthetic context adaptation using only five example documents performs within 0.5% of models leveraging full target corpus access -- demonstrating remarkable efficacy without any retraining. ZEST thus provides a practical method for deploying high-performance, adaptable embeddings in constrained environments.

Related papers

• GrOIL: Graph-Grounded Domain Ontology Induction with Constrained LLM Mediation [1.029029220360574]
  We present a seven-stage graph-grounded pipeline that converts domain documents into a complete Web Ontology Language (OWL) Terminological Box (TBox) without any unconstrained generation step.<n>Ontology growth analysis provides evidence consistent with vocabulary saturation at scale, demonstrating that the pipeline produces stable, reusable domain representations from large document corpora.
  arXiv  Detail & Related papers  (2026-08-22T23:15:47Z)
• Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction [127.64173950476702]
  We study direct corpus interaction (DCI), where an agent searches the raw corpus directly with general-purpose terminal tools.<n>This approach requires no offline indexing and adapts naturally to evolving local corpora.<n>Across IR benchmarks and end-to-end agentic search tasks, this simple setup substantially outperforms strong sparse, dense, and reranking baselines.
  arXiv  Detail & Related papers  (2026-05-03T19:13:11Z)
• Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training [23.87978106727431]
  We propose a data-training co-design framework for robust end-to-end document parsing.<n>Our method achieves superior accuracy and robustness across both scanned/digital and real-world captured scenarios.
  arXiv  Detail & Related papers  (2026-03-25T03:19:09Z)
• CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning [48.56088080889236]
  We introduce CorpusQA, a new benchmark scaling up to 10 million tokens, generated via a novel data synthesis framework.<n>We show that finetuning on our synthesized data effectively enhances an LLM's general long-context reasoning capabilities.<n>Our findings indicate that memory-augmented agentic architectures offer a more robust alternative.
  arXiv  Detail & Related papers  (2026-01-21T12:52:30Z)
• FeClustRE: Hierarchical Clustering and Semantic Tagging of App Features from User Reviews [0.0]
  FeClustRE is a framework integrating hybrid feature extraction, hierarchical clustering with auto-tuning and semantic labelling.<n>We evaluate FeClustRE on public benchmarks for extraction correctness and on a sample study of generative AI assistant app reviews for clustering quality, semantic coherence, and interpretability.
  arXiv  Detail & Related papers  (2025-10-21T16:54:21Z)
• LLM-guided Hierarchical Retrieval [54.73080745446999]
  LATTICE is a hierarchical retrieval framework that enables an LLM to reason over and navigate large corpora with logarithmic search complexity.<n>A central challenge in such LLM-guided search is that the model's relevance judgments are noisy, context-dependent, and unaware of the hierarchy.<n>Our framework achieves state-of-the-art zero-shot performance on the reasoning-intensive BRIGHT benchmark.
  arXiv  Detail & Related papers  (2025-10-15T07:05:17Z)
• RecBase: Generative Foundation Model Pretraining for Zero-Shot Recommendation [78.01030342481246]
  RecBase is a domain-agnostic foundational model pretrained with a recommendation-oriented objective.<n>We introduce a unified item tokenizer that encodes items into hierarchical concept identifiers.<n>Our model matches or surpasses the performance of LLM baselines up to 7B parameters in zero-shot and cross-domain recommendation tasks.
  arXiv  Detail & Related papers  (2025-09-03T08:33:43Z)
• ABCD-LINK: Annotation Bootstrapping for Cross-Document Fine-Grained Links [57.514511353084565]
  We introduce a new domain-agnostic framework for selecting a best-performing approach and annotating cross-document links.<n>We apply our framework in two distinct domains -- peer review and news.<n>The resulting novel datasets lay foundation for numerous cross-document tasks like media framing and peer review.
  arXiv  Detail & Related papers  (2025-09-01T11:32:24Z)
• Multi-Relation Extraction in Entity Pairs using Global Context [0.8437187555622164]
  This paper introduces a novel input embedding approach to capture the positions of mentioned entities throughout a document.<n>The performance of the proposed method has been tested on three benchmark relation extraction datasets.<n>Theoretically, it advances global context modeling and multi-sentence reasoning in document-level relation extraction.
  arXiv  Detail & Related papers  (2025-07-23T13:24:32Z)
• Class-Agnostic Region-of-Interest Matching in Document Images [5.0512633844625405]
  This paper defines a new task named Class-Agnostic Region-of-Interest Matching''<n>It aims to match customized regions in a flexible, efficient, multi-granularity, and open-set manner.<n>We construct a benchmark RoI-Matching-Bench, which sets three levels of difficulties following real-world conditions.<n>We also propose a new framework RoI-Matcher, which employs a siamese network to extract multi-level features.
  arXiv  Detail & Related papers  (2025-06-26T07:09:19Z)
• Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings [25.966475857117175]
  In this work, we introduce ConTEB, a benchmark designed to evaluate retrieval models on their ability to leverage document-wide context.<n>Our results show that state-of-the-art embedding models struggle in retrieval scenarios where context is required.<n>We propose InSeNT, a novel contrastive post-training approach which combined with late chunking pooling enhances contextual representation learning.
  arXiv  Detail & Related papers  (2025-05-30T16:43:28Z)
• Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentation [72.28364940168092]
  Open-vocabulary semantic segmentation models associate vision and text to label pixels from an undefined set of classes using textual queries.<n>We introduce Semantic Library Adaptation (SemLA), a novel framework for training-free, test-time domain adaptation.
  arXiv  Detail & Related papers  (2025-03-27T17:59:58Z)
• Contextual Document Embeddings [77.22328616983417]
  We propose two complementary methods for contextualized document embeddings. First, an alternative contrastive learning objective that explicitly incorporates the document neighbors into the intra-batch contextual loss. Second, a new contextual architecture that explicitly encodes neighbor document information into the encoded representation.
  arXiv  Detail & Related papers  (2024-10-03T14:33:34Z)
• Text-Video Retrieval with Global-Local Semantic Consistent Learning [122.15339128463715]
  We propose a simple yet effective method, Global-Local Semantic Consistent Learning (GLSCL) GLSCL capitalizes on latent shared semantics across modalities for text-video retrieval. Our method achieves comparable performance with SOTA as well as being nearly 220 times faster in terms of computational cost.
  arXiv  Detail & Related papers  (2024-05-21T11:59:36Z)
• CREPE: Coordinate-Aware End-to-End Document Parser [13.530212337717515]
  We formulate an OCR-free sequence generation model for visual document understanding (VDU) Our model not only parses text from document images but also extracts the spatial coordinates of the text based on the multi-head architecture. Named as Coordinate-aware End-to-end Document. CREPE, our method uniquely integrates these capabilities by introducing a special token for OCR text.
  arXiv  Detail & Related papers  (2024-05-01T00:30:13Z)
• TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
  We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
  arXiv  Detail & Related papers  (2023-06-06T03:37:41Z)
• Top-Down Synthesis for Library Learning [46.285220926554345]
  corpus-guided top-down synthesis is a mechanism for synthesizing library functions that capture common functionality from a corpus of programs. We present an implementation of the approach in a tool called Stitch and evaluate it against the state-of-the-art deductive library learning algorithm from DreamCoder.
  arXiv  Detail & Related papers  (2022-11-29T21:57:42Z)
• Beyond the Prototype: Divide-and-conquer Proxies for Few-shot Segmentation [63.910211095033596]
  Few-shot segmentation aims to segment unseen-class objects given only a handful of densely labeled samples. We propose a simple yet versatile framework in the spirit of divide-and-conquer. Our proposed approach, named divide-and-conquer proxies (DCP), allows for the development of appropriate and reliable information.
  arXiv  Detail & Related papers  (2022-04-21T06:21:14Z)
• Extractive Summarization as Text Matching [123.09816729675838]
  This paper creates a paradigm shift with regard to the way we build neural extractive summarization systems. We formulate the extractive summarization task as a semantic text matching problem. We have driven the state-of-the-art extractive result on CNN/DailyMail to a new level (44.41 in ROUGE-1)
  arXiv  Detail & Related papers  (2020-04-19T08:27:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.