Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Search

• URL: http://arxiv.org/abs/2510.22101v1
• Date: Sat, 25 Oct 2025 00:56:06 GMT
• Title: Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Search
• Abstract summary: Large Language Models (LLMs) have demonstrated impressive quality when applied to predictive tasks such as relevance ranking and semantic search.<n>However, deployment of such LLMs remains prohibitively expensive for industry applications with strict latency and throughput requirements.<n>We present lessons and efficiency insights from developing a purely text-based decoder-only Small Language Model (SLM) for a semantic search application at LinkedIn.
• Score: 12.891315513255527
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) have demonstrated impressive quality when applied to predictive tasks such as relevance ranking and semantic search. However, deployment of such LLMs remains prohibitively expensive for industry applications with strict latency and throughput requirements. In this work, we present lessons and efficiency insights from developing a purely text-based decoder-only Small Language Model (SLM) for a semantic search application at LinkedIn. Particularly, we discuss model compression techniques such as pruning that allow us to reduce the model size by up to $40\%$ while maintaining the accuracy. Additionally, we present context compression techniques that allow us to reduce the input context length by up to $10$x with minimal loss of accuracy. Finally, we present practical lessons from optimizing the serving infrastructure for deploying such a system on GPUs at scale, serving millions of requests per second. Taken together, this allows us to increase our system's throughput by $10$x in a real-world deployment, while meeting our quality bar.

Related papers

• Stacked from One: Multi-Scale Self-Injection for Context Window Extension [69.24689919827817]
  modelname is a novel framework based on multi-grained context compression and query-aware information acquisition.<n>modelnameachieves performance superior or comparable to strong baselines.
  arXiv  Detail & Related papers  (2026-03-05T03:16:16Z)
• Less is More: Adapting Text Embeddings for Low-Resource Languages with Small Scale Noisy Synthetic Data [0.0]
  Low-resource languages (LRLs) often lack high-quality, large-scale datasets for training effective text embedding models.<n>We introduce a cost-effective adaptation strategy using small scale noisy synthetic data generated by translating English Reddit title-body pairs.<n>Experiments reveal a surprising "Less is More" phenomenon: fine-tuning a multilingual encoder on just 10,000 noisy synthetic pairs yields 11-12% average improvements.
  arXiv  Detail & Related papers  (2026-03-04T07:19:51Z)
• AgentCompress: Task-Aware Compression for Affordable Large Language Model Agents [0.0]
  A single session using a 70-billion- parameter model can cost around $127 in cloud computing fees.<n>We present AgentCompress, a framework that tackles this problem through task-aware dynamic compression.<n>Results show a 68.3% reduction in computational costs while preserving 96.2% of the original success rate.
  arXiv  Detail & Related papers  (2026-01-08T18:13:46Z)
• Language Ranker: A Lightweight Ranking framework for LLM Decoding [70.01564145836129]
  This paper conceptualizes the decoding process as analogous to the ranking stage in recommendation pipelines.<n>Motivated by this insight, we propose Language Ranker, a novel framework that introduces a lightweight module to rerank candidate responses.<n> Experiments show that Language Ranker achieves performance comparable to large-scale reward models, while requiring only 0.5M additional parameters.
  arXiv  Detail & Related papers  (2025-10-23T17:56:46Z)
• Compressing Large Language Models with Automated Sub-Network Search [41.452512557226335]
  We consider model compression for Large Language Models to reduce model size while improving downstream task performance.<n>We phrase this as a neural architecture search problem that automatically prunes structural components.<n>Our method achieves upto 9.85% improvement on average on 11 diverse downstream tasks, while achieving up to 22% improvement of on-device latency.
  arXiv  Detail & Related papers  (2024-10-09T02:14:39Z)
• Mixture of Attentions For Speculative Decoding [17.344416130742232]
  Speculative decoding (SD) leverages smaller models to efficiently propose future tokens, which are then verified by the Large Language Models in parallel.<n>We identify several limitations of SD models including the lack of on-policyness during training and partial observability.<n>We propose a more grounded architecture for small models by introducing a Mixture of Attentions for SD.
  arXiv  Detail & Related papers  (2024-10-04T10:25:52Z)
• Aggressive Post-Training Compression on Extremely Large Language Models [32.589344168888914]
  Aggressive post-training model compression is necessary to reduce the models' size. We propose a novel network pruning technology that utilizes over 0.7 sparsity and less than 8 bits of quantization.
  arXiv  Detail & Related papers  (2024-09-30T08:47:17Z)
• LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models [22.06402870816756]
  Large language models (LLMs) have been applied in various applications due to their astonishing capabilities. This paper presents LLMLingua, a coarse-to-fine prompt compression method that involves a budget controller to maintain semantic integrity. We show that the proposed approach yields state-of-the-art performance and allows for up to 20x compression with little performance loss.
  arXiv  Detail & Related papers  (2023-10-09T14:10:21Z)
• Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt [96.24800696597707]
  We introduce a new perspective to optimize this trade-off by prompting compressed models. We propose a soft prompt learning method where we expose the compressed model to the prompt learning process. Our experimental analysis suggests our soft prompt strategy greatly improves the performance of the 8x compressed LLaMA-7B model.
  arXiv  Detail & Related papers  (2023-05-17T20:45:13Z)
• Speculative Decoding with Big Little Decoder [108.95187338417541]
  Big Little Decoder (BiLD) is a framework that can improve inference efficiency and latency for a wide range of text generation applications. On an NVIDIA T4 GPU, our framework achieves a speedup of up to 2.12x speedup with minimal generation quality degradation. Our framework is fully plug-and-play and can be applied without any modifications in the training process or model architecture.
  arXiv  Detail & Related papers  (2023-02-15T18:55:29Z)
• ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training [65.68511423300812]
  We propose ProgFed, a progressive training framework for efficient and effective federated learning. ProgFed inherently reduces computation and two-way communication costs while maintaining the strong performance of the final models. Our results show that ProgFed converges at the same rate as standard training on full models.
  arXiv  Detail & Related papers  (2021-10-11T14:45:00Z)
• CPM-2: Large-scale Cost-effective Pre-trained Language Models [71.59893315671997]
  We present a suite of cost-effective techniques for the use of PLMs to deal with the efficiency issues of pre-training, fine-tuning, and inference. We introduce knowledge inheritance to accelerate the pre-training process by exploiting existing PLMs instead of training models from scratch. We implement a new inference toolkit, namely InfMoE, for using large-scale PLMs with limited computational resources.
  arXiv  Detail & Related papers  (2021-06-20T15:43:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.