Benchmarking Patent Embeddings: A Multi-Task Evaluation of 22 Models Across Retrieval, Classification, and Clustering

• URL: http://arxiv.org/abs/2605.24297v2
• Date: Tue, 26 May 2026 04:08:36 GMT
• Title: Benchmarking Patent Embeddings: A Multi-Task Evaluation of 22 Models Across Retrieval, Classification, and Clustering
• Abstract summary: We evaluate 22 pre-trained embedding models on three tasks -- information retrieval, classification, and clustering.<n>We find that two results cast doubt on the prevailing wisdom.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Two questions regarding practitioners' use of patent embeddings arise: (i) Does one fine-tuning recipe suffice for all downstream applications? (ii) Is fine-tuning on one patent landscape sufficient for downstream application on other landscapes? By evaluating 22 pre-trained embedding models (ranging from 22M to 12B parameters) on three tasks -- information retrieval, classification, and clustering -- on 113,148 WIPO patents for assistive technology (46,069 citation queries) and on an external DAPFAM dataset, we find that two results cast doubt on the prevailing wisdom. (i) The optimal fine-tuning recipe depends on the downstream task: cross-sectional alignment (recipe R3) provides the largest improvements to retrieval performance (+7.1% nDCG@10), whereas a combined signal recipe (recipe R4) is better suited to classification (+7.1 F1) and clustering (+10.9 V-measure); a matched data control confirms that differences in training dataset size are not a contributing factor. (ii) Single-landscape fine-tuning hampers cross-landscape information retrieval: fine-tuning on one landscape significantly degrades cross-domain retrieval for 5 of 8 model-recipe combinations on the DAPFAM corpus, with the stronger zero-shot models suffering most. While within-family scaling is consistent (Qwen3 0.6B->4B->8B; Llama-Nemotron 1B->8B), cross-family scaling is erratic; the 12B KaLM-Gemma3 is ranked 8th on TAC retrieval performance, following prefix modification. Title+Abstract+Claims is the ubiquitous best text view, and all models suffer from a 55-65% gap between IN and OUT-of-domain performance which cannot be mitigated by hybrid BM25-dense fusion. Code and evaluation framework are publicly available.

Related papers

• RecGPT-Mobile-V2 Technical Report [22.10290138930905]
  RecGPT-Mobile-V2 is an end-to-end framework that treats intent quality and execution efficiency as coupled objectives within a staged design.<n>The framework transforms heterogeneous interactions into an evidence-preserving trajectory, establishes a recommendation-native foundation through domain adaptation and supervised alignment.<n>Online retrieval analysis indicates that the Query recall channel retrieves inventory complementary to that surfaced by established recall channels.
  arXiv  Detail & Related papers  (2026-08-25T09:23:17Z)
• Towards Robust Training in NNGPT AutoML Pipeline: A Loss-Optimizer Pairing Selection Study [48.83701310501069]
  This paper investigates whether a single recipe is sufficient for heterogeneous architecture pools, or whether the optimal pairing varies across structurally diverse models.<n>We conduct a systematic empirical study of all $3 times 6 = 18$ combinations of six classifiers paired with three loss functions: Cross-Entropy (CEL), Negative Log-Likelihood (NLL), and the recently introduced genetically evolved NGL loss across the base models presented in LEMUR heterogeneous architecture pool on six image classification datasets.<n>Our results confirm that no single pairing is universally optimal. Cross-Entropy with Adam or AdamW is the most robust choice across
  arXiv  Detail & Related papers  (2026-06-18T20:51:42Z)
• How Much Capacity Does EEG Denoising Need? Ultra-Compact Networks reveal Benchmark Saturation and Metric-Utility Gap [0.0]
  Deep learning EEG denoising architectures have scaled from tens of thousands to tens of millions of parameters, yet no prior study has isolated model capacity as the experimental variable.<n>We address both gaps by fixing architecture, loss, data split, and training recipe while sweeping only channel width from 1.05K to 40.26K parameters in a minimal depthwiseseparable convolutional U-Net.
  arXiv  Detail & Related papers  (2026-06-07T12:17:25Z)
• Domain-Adapted Small Language Models with Hybrid Post-Processing: Achieving Cost-Efficient, Low-Latency Multi-Label Structured Prediction via LoRA Fine-Tuning on Scarce Data [6.3745740668603075]
  Large language models (LLMs) for domain-specific structured evaluation tasks incurs prohibitive latency, cost, and data-privacy overhead.<n>We present a hybrid framework that fine-tunes a small language model on only 219 curated examples and couples it with a deterministic rule-based postprocessing layer.<n>Our system achieves 100% structural validity, 83.0% human-validated overall accuracy, and 100% accuracy on the most critical classification field in blind evaluation on 53 unseen production transcripts.
  arXiv  Detail & Related papers  (2026-06-04T07:09:37Z)
• When Does Synthetic Patent Data Help? Volume-Fidelity Trade-offs in Low-Resource Multi-Label Classification [0.0]
  It is shown that the claimed improvements in micro F1 for BERT-for-Patents mainly reflect a volume effect.<n>A corpus that allows for improvement in classification performance up to +0.58 in raw micro F1 may adversely affect a Jaccard-overlap retrieval proxy.
  arXiv  Detail & Related papers  (2026-05-22T23:49:52Z)
• Models Can Model, But Can't Bind: Structured Grounding in Text-to-Optimization [54.749573452394664]
  We find that accuracy collapses as instance data grows, even when the formulation itself is simple.<n>We address this via a simple inference-time approach, BIND, which externalizes numeric data to structured files so the model binds data rather than from prompt prompt.<n>We validate our hypothesis by finetuning a model exclusively on binding and show that it outperforms end-to-end SFT and RL across three structurally distinct optimization categories.
  arXiv  Detail & Related papers  (2026-05-20T21:25:41Z)
• PRUE: A Practical Recipe for Field Boundary Segmentation at Scale [50.194423500109025]
  We conduct the first systematic evaluation of segmentation and geospatial foundation models (GFMs) for global field boundary delineation.<n>A U-Net semantic segmentation model outperforms instance-based and GFM alternatives on a suite of performance and deployment metrics.<n>Our approach provides a practical framework for reliable, scalable, and reproducible field boundary delineation across model design, training, and inference.
  arXiv  Detail & Related papers  (2026-03-28T02:47:46Z)
• Assessing LLM Reliability on Temporally Recent Open-Domain Questions [15.456770184839726]
  Large Language Models (LLMs) are increasingly deployed for open-domain question answering.<n>We investigate how four open-source LLMs respond to 15,000 recent Reddit questions.<n>All models achieve over 99% cosine similarity with references despite less than 8% BLEU-1 overlap.
  arXiv  Detail & Related papers  (2026-01-17T21:33:27Z)
• Large-Scale Aspect-Based Sentiment Analysis with Reasoning-Infused LLMs [1.4732811715354455]
  Arctic-ABSA is a collection of powerful models for real-life aspect-based sentiment analysis (ABSA)<n>Our models are tailored to commercial needs, trained on a large corpus of public data alongside carefully generated synthetic data, resulting in a dataset 20 times larger than SemEval14.<n>A single multilingual model maintains 87-91% accuracy across six languages without degrading English performance.
  arXiv  Detail & Related papers  (2026-01-07T13:58:29Z)
• EdgeJury: Cross-Reviewed Small-Model Ensembles for Truthful Question Answering on Serverless Edge Inference [0.0]
  We present EdgeJury, a lightweight ensemble framework that improves truthfulness and robustness.<n>On TruthfulQA (MC1), EdgeJury achieves 76.2% accuracy.<n>On a 200-question adversarial EdgeCases set, EdgeJury yields +48.2% relative gains.
  arXiv  Detail & Related papers  (2025-12-29T14:48:40Z)
• PatenTEB: A Comprehensive Benchmark and Model Family for Patent Text Embedding [1.0742675209112622]
  Patent text embeddings enable prior art search, technology landscaping, and patent analysis, yet existing benchmarks inadequately capture patent-specific challenges.<n>We introduce PatenTEB, a comprehensive benchmark comprising 15 tasks across retrieval, classification, paraphrase, and clustering, with 2.06 million examples.<n>We develop the patembed model family through multi-task training, spanning 67M to 344M parameters with context lengths up to 4096 tokens.
  arXiv  Detail & Related papers  (2025-10-25T12:01:46Z)
• David vs. Goliath: A comparative study of different-sized LLMs for code generation in the domain of automotive scenario generation [1.6752458252726459]
  NL-to-Scenic generation with large language models (LLMs) suffers from scarce data, limited metrics.<n>We introduce NL2Scenic, an open dataset and framework with 146 NL/Scenic pairs, a difficulty-stratified 30-case test split, and an Example Retriever.<n>We evaluate 13 models: four proprietary (GPT-4o, GPT-5, Claude-Sonnet-4, Gemini-2.5-pro) and nine open-source code models (Qwen2.5Coder 0.5B-32B; CodeLlama 7B/13B/34B)
  arXiv  Detail & Related papers  (2025-10-15T21:37:02Z)
• LLM Routing with Dueling Feedback [49.67815163970033]
  We study the problem of selecting the best model for each query while balancing user satisfaction, model expertise, and inference cost.<n>We formulate routing as contextual dueling bandits, learning from pairwise preference feedback rather than absolute scores.<n>We introduce Category-Calibrated Fine-Tuning (CCFT), a representation-learning method that derives model embeddings from offline data using contrastive fine-tuning with categorical weighting.
  arXiv  Detail & Related papers  (2025-10-01T12:52:25Z)
• AHDMIL: Asymmetric Hierarchical Distillation Multi-Instance Learning for Fast and Accurate Whole-Slide Image Classification [51.525891360380285]
  AHDMIL is an Asymmetric Hierarchical Distillation Multi-Instance Learning framework.<n>It eliminates irrelevant patches through a two-step training process.<n>It consistently outperforms previous state-of-the-art methods in both classification performance and inference speed.
  arXiv  Detail & Related papers  (2025-08-07T07:47:16Z)
• S*: Test Time Scaling for Code Generation [55.11863577956177]
  We propose S*, the first hybrid test-time scaling framework for code generation.<n>S* substantially improves the coverage and selection accuracy of generated code.
  arXiv  Detail & Related papers  (2025-02-20T09:18:53Z)
• Towards Generalizable Trajectory Prediction Using Dual-Level Representation Learning And Adaptive Prompting [107.4034346788744]
  Existing vehicle trajectory prediction models struggle with generalizability, prediction uncertainties, and handling complex interactions.<n>We propose Perceiver with Register queries (PerReg+), a novel trajectory prediction framework that introduces: (1) Dual-Level Representation Learning via Self-Distillation (SD) and Masked Reconstruction (MR), capturing global context and fine-grained details; (2) Enhanced Multimodality using register-based queries and pretraining, eliminating the need for clustering and suppression; and (3) Adaptive Prompt Tuning during fine-tuning, freezing the main architecture and optimizing a small number of prompts for efficient adaptation.
  arXiv  Detail & Related papers  (2025-01-08T20:11:09Z)
• Text Classification via Large Language Models [63.1874290788797]
  We introduce Clue And Reasoning Prompting (CARP) to address complex linguistic phenomena involved in text classification. Remarkably, CARP yields new SOTA performances on 4 out of 5 widely-used text-classification benchmarks. More importantly, we find that CARP delivers impressive abilities on low-resource and domain-adaptation setups.
  arXiv  Detail & Related papers  (2023-05-15T06:24:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.