KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling

• URL: http://arxiv.org/abs/2607.09153v1
• Date: Fri, 10 Jul 2026 07:16:43 GMT
• Title: KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling
• Abstract summary: Process Reward Models (PRMs) are proven to be highly effective in guiding test-time scaling (TTS) methods.<n>Existing PRMs are text-based: they re-encode the entire trajectory text from scratch.<n>In long multi-agent rollouts, the scoring cost, growing quadratically with respect to sequence length L, creates a severe computational bottleneck.<n>We introduce KV-PRM, a highly efficient process reward model that eliminates the heavy text re-encoding by directly reading the KV cache.
• Score: 43.43702580827712
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Process Reward Models (PRMs) have been proven to be highly effective in guiding test-time scaling (TTS) methods, which significantly boost the capabilities of LLM-based multi-agent systems. However, existing PRMs are text-based: they re-encode the entire trajectory text from scratch. In long multi-agent rollouts, the scoring cost, growing quadratically with respect to sequence length L, creates a severe computational bottleneck, severely limiting PRMs' application in long-context scenarios. To resolve this, we introduce KV-PRM, a highly efficient process reward model that eliminates the heavy text re-encoding by directly reading the KV cache produced naturally during the LLM's generation phase. By processing a single "verify token" against the pre-existing KV cache, KV-PRM reduces the scoring cost from O(L^2) to O(L). We formally prove that the KV cache contains strictly greater information capacity than text, and is more efficient for downstream reward modeling. Empirically, across the MATH, GSM8K, and AIME benchmarks, KV-PRM matches or strictly outperforms text-PRMs under various TTS methods such as Beam Search, MCTS, and Weighted Voting, with up to a 5,000x reduction in scoring FLOPs, a 37x reduction in latency, and a 34x reduction in per-sequence memory footprint compared to text-based PRMs.

Related papers

• Tiny Inference-Time Scaling with Latent Verifiers [56.696619768584675]
  Verifier on Hidden States ( VHS) operates on intermediate hidden representations of Diffusion Transformer (DiT) single-step generators.<n> VHS analyzes generator features without decoding to pixel space, thereby reducing the per-candidate verification cost.<n> VHS achieves a +2.7% improvement on GenEval at the same inference-time budget.
  arXiv  Detail & Related papers  (2026-03-23T19:00:02Z)
• RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse [5.597099794399441]
  RelayCaching is a training-free inference method that directly reuses decoding phase KV caches from previous agents.<n>We show that RelayCaching achieves over 80% KV cache reuse, reduces TTFT by up to $4.7times$ compared to the standard pipeline.
  arXiv  Detail & Related papers  (2026-02-28T04:46:28Z)
• ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs [1.1267872663780352]
  Large Language Models (LLMs) are increasingly deployed in scenarios demanding ultra-long context reasoning.<n>Existing memory reduction techniques, including eviction and quantization, often rely on static cache.<n>We propose ARKV, a lightweight and adaptive framework that dynamically allocates precision levels to cached tokens.
  arXiv  Detail & Related papers  (2026-02-19T16:24:08Z)
• DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity [50.52392445266824]
  We propose a residual-based KV cache compression framework motivated by long-range inter-token similarity and highly shared latent components in KV representations.<n>Instead of discarding tokens, DeltaKV encodes semantic residuals relative to retrieved historical references, preserving fidelity while substantially reducing storage.<n>Experiments show that DeltaKV reduces KV cache memory to 29% of the original while maintaining near-lossless accuracy on LongBench, SCBench, and AIME.
  arXiv  Detail & Related papers  (2026-02-08T15:14:36Z)
• SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Models [25.509962883211]
  Large reasoning models (LRMs) often cost significant key-value (KV) cache overhead, due to their linear growth with the chain-of-thought (CoT) reasoning process.<n>We present textbfSkipKV, a KV compression method for selective textiteviction and textitgeneration operating at a coarse-grained sentence-level sequence removal.
  arXiv  Detail & Related papers  (2025-12-08T19:32:06Z)
• R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoning [80.104336426172]
  Chain-of-thought (CoT) enhances problem-solving ability of large language models.<n>CoT incurs substantial inference cost due to long autoregressive trajectories.<n>We introduce R-Stitch, a training-free hybrid decoding framework.
  arXiv  Detail & Related papers  (2025-07-23T08:14:36Z)
• Accelerating Diffusion LLMs via Adaptive Parallel Decoding [60.407727995313074]
  We introduce adaptive parallel decoding (APD), a novel method that dynamically adjusts the number of tokens sampled in parallel.<n>APD provides markedly higher throughput with minimal quality degradations on downstream benchmarks.
  arXiv  Detail & Related papers  (2025-05-31T06:10:10Z)
• HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference [24.068304021577358]
  Disaggregated Large Language Model (LLM) inference separates computation-intensive prefill stage from memory-intensive decode stage.<n> transmitting Key-Value (KV) data between the two stages can be a bottleneck, especially for long prompts.<n>We propose Homomorphic Acceleration via Compression of the KV cache (HACK) for disaggregated LLM inference.
  arXiv  Detail & Related papers  (2025-02-05T20:09:51Z)
• LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy [59.1298692559785]
  Key-Value ( KV) cache is crucial component in serving transformer-based autoregressive large language models (LLMs) Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages; (2) KV cache compression at test time; and (3) KV cache compression at test time. We propose a low-rank approximation of KV weight matrices, allowing plug-in integration with existing transformer-based LLMs without model retraining. Our method is designed to function without model tuning in upcycling stages or task-specific profiling in test stages.
  arXiv  Detail & Related papers  (2024-10-04T03:10:53Z)
• ThinK: Thinner Key Cache by Query-Driven Pruning [63.13363917871414]
  Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications.<n>This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference.<n>We propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels.
  arXiv  Detail & Related papers  (2024-07-30T17:59:08Z)
• Effectively Compress KV Heads for LLM [28.0801697946958]
  We propose a novel approach for compressing Key-Value ( KV) caches. Our method can compress half or even three-quarters of KV heads while maintaining performance comparable to the original LLMs.
  arXiv  Detail & Related papers  (2024-06-11T08:37:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.