From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings

• URL: http://arxiv.org/abs/2603.03301v1
• Date: Sat, 07 Feb 2026 10:16:46 GMT
• Title: From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings
• Abstract summary: Large language models (LLMs) have created demand for faster responses and lower costs.<n>Semantic caching, reusing semantically similar requests via their embeddings, addresses this need but breaks classic cache assumptions.<n>We present online semantic cache policies that combine recency, frequency, and locality.
• Score: 2.3204178451683264
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid adoption of large language models (LLMs) has created demand for faster responses and lower costs. Semantic caching, reusing semantically similar requests via their embeddings, addresses this need but breaks classic cache assumptions and raises new challenges. In this paper, we explore offline policies for semantic caching, proving that implementing an optimal offline policy is NP-hard, and propose several polynomial-time heuristics. We also present online semantic aware cache policies that combine recency, frequency, and locality. Evaluations on diverse datasets show that while frequency based policies are strong baselines, our novel variant improves semantic accuracy. Our findings reveal effective strategies for current systems and highlight substantial headroom for future innovation. All code is open source.

Related papers

• Asynchronous Verified Semantic Caching for Tiered LLM Architectures [0.7204795910838664]
  Large language models (LLMs) now sit in the critical path of search, assistance, and agentic.<n>Production deployments typically use a tiered static-dynamic design: a static cache of curated, offline vetted responses mined from logs, backed by a dynamic cache populated online.<n>We introduce textbfKrites, an asynchronous, LLM-judged caching policy that expands static coverage without changing serving decisions.
  arXiv  Detail & Related papers  (2026-02-13T18:25:00Z)
• Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online Adaptation [54.61034867177997]
  Caching inference responses allows them to be retrieved without another forward pass through the Large Language Models.<n>Traditional exact-match caching overlooks the semantic similarity between queries, leading to unnecessary recomputation.<n>We present a principled, learning-based framework for semantic cache eviction under unknown query and cost distributions.
  arXiv  Detail & Related papers  (2025-08-11T06:53:27Z)
• LoopServe: An Adaptive Dual-phase LLM Inference Acceleration System for Multi-Turn Dialogues [59.12542274007847]
  LoopServe is an adaptive dual-phase inference framework for large language models in multi-turn dialogues.<n>LoopServe consistently achieves superior effectiveness compared to existing baselines.
  arXiv  Detail & Related papers  (2025-07-18T06:12:08Z)
• ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models [67.75439511654078]
  Large Vision-Language Models (LVLMs) have introduced a new paradigm for understanding and reasoning about image input through textual responses.<n>They face the persistent challenge of hallucination, which introduces practical weaknesses and raises concerns about their reliable deployment in real-world applications.<n>We propose ONLY, a training-free decoding approach that requires only a single query and a one-layer intervention during decoding, enabling efficient real-time deployment.
  arXiv  Detail & Related papers  (2025-07-01T16:01:08Z)
• Edge Caching Optimization with PPO and Transfer Learning for Dynamic Environments [3.720975664058743]
  In dynamic environments, changes in content popularity and variations in request rates frequently occur, making previously learned policies less effective as they were optimized for earlier conditions. We develop a mechanism that detects changes in content popularity and request rates, ensuring timely adjustments to the caching strategy. We also propose a transfer learning-based PPO algorithm that accelerates convergence in new environments by leveraging prior knowledge.
  arXiv  Detail & Related papers  (2024-11-14T21:01:29Z)
• Efficient Inference of Vision Instruction-Following Models with Elastic Cache [76.44955111634545]
  We introduce Elastic Cache, a novel strategy for efficient deployment of instruction-following large vision-language models. We propose an importance-driven cache merging strategy to prune redundancy caches. For instruction encoding, we utilize the frequency to evaluate the importance of caches. Results on a range of LVLMs demonstrate that Elastic Cache not only boosts efficiency but also notably outperforms existing pruning methods in language generation.
  arXiv  Detail & Related papers  (2024-07-25T15:29:05Z)
• An Online Gradient-Based Caching Policy with Logarithmic Complexity and Regret Guarantees [13.844896723580858]
  We introduce a new variant of the gradient-based online caching policy that achieves groundbreaking logarithmic computational complexity. This advancement allows us to test the policy on large-scale, real-world traces featuring millions of requests and items.
  arXiv  Detail & Related papers  (2024-05-02T13:11:53Z)
• Temporal-aware Hierarchical Mask Classification for Video Semantic Segmentation [62.275143240798236]
  Video semantic segmentation dataset has limited categories per video. Less than 10% of queries could be matched to receive meaningful gradient updates during VSS training. Our method achieves state-of-the-art performance on the latest challenging VSS benchmark VSPW without bells and whistles.
  arXiv  Detail & Related papers  (2023-09-14T20:31:06Z)
• Cocktail Edge Caching: Ride Dynamic Trends of Content Popularity with Ensemble Learning [10.930268276150262]
  Edge caching will play a critical role in facilitating the emerging content-rich applications. It faces many new challenges, in particular, the highly dynamic content popularity and the heterogeneous caching computation. We propose Cocktail Edge Caching, that tackles the dynamic popularity and heterogeneity through ensemble learning.
  arXiv  Detail & Related papers  (2021-01-14T21:59:04Z)
• Reinforcement Learning for Caching with Space-Time Popularity Dynamics [61.55827760294755]
  caching is envisioned to play a critical role in next-generation networks. To intelligently prefetch and store contents, a cache node should be able to learn what and when to cache. This chapter presents a versatile reinforcement learning based approach for near-optimal caching policy design.
  arXiv  Detail & Related papers  (2020-05-19T01:23:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.