Quantized Inference for OneRec-V2

• URL: http://arxiv.org/abs/2603.11486v1
• Date: Thu, 12 Mar 2026 03:13:08 GMT
• Title: Quantized Inference for OneRec-V2
• Abstract summary: We develop a FP8 post training quantization framework and integrate it into an optimized inference infrastructure.<n>The proposed joint optimization achieves a 49% reduction in end-to-end inference latency and a 92% increase in throughput.<n>These results suggest that as recommender systems evolve toward the paradigms of large language models, algorithm-level and system-level optimization techniques can be effectively adapted to large-scale recommendation workloads.
• Score: 28.8042155916846
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Quantized inference has demonstrated substantial system-level benefits in large language models while preserving model quality. In contrast, reliably applying low-precision quantization to recommender systems remains challenging in industrial settings. This difficulty arises from differences in training paradigms, architectural patterns, and computational characteristics, which lead to distinct numerical behaviors in weights and activations. Traditional recommender models often exhibit high-magnitude and high-variance weights and activations, making them more sensitive to quantization-induced perturbations. In addition, recommendation workloads frequently suffer from limited hardware utilization, limiting the practical gains of low-precision computation. In this work, we revisit low-precision inference in the context of generative recommendation. Through empirical distribution analysis, we show that the weight and activation statistics of OneRec-V2 are significantly more controlled and closer to those of large language models than traditional recommendation models. Moreover, OneRec-V2 exhibits a more compute-intensive inference pattern with substantially higher hardware utilization, enabling more end-to-end throughput gains with low-precision computation. Leveraging this property, we develop a FP8 post training quantization framework and integrate it into an optimized inference infrastructure. The proposed joint optimization achieves a 49\% reduction in end-to-end inference latency and a 92\% increase in throughput. Extensive online A/B testing further confirms that FP8 inference introduces no degradation in core metrics. These results suggest that as recommender systems evolve toward the paradigms of large language models, algorithm-level and system-level optimization techniques established in the LLM domain can be effectively adapted to large-scale recommendation workloads.

Related papers

• WINDQuant: Weight-Informed Neural Decision-Making for Global Mixed-Precision LLM Quantization [40.655670203062805]
  WINDQuant is a reinforcement-learning-based allocation controller for ultra-low-bit LLM quantization.<n>It learns how to assign bit-widths and quantization treatments to fine-grained column chunks under a global storage budget.<n>Experiments on LLaMA models demonstrate that WINDQuant achieves competitive performance in ultra-low-bit settings.
  arXiv  Detail & Related papers  (2026-05-26T07:46:13Z)
• ECO: Quantized Training without Full-Precision Master Weights [58.97082407934466]
  Error-Compensating (ECO) eliminates master weights by applying updates directly to quantized parameters.<n>We show that ECO converges to a constant-radius neighborhood of the optimum, while naive master-weight removal can incur an error that is inversely proportional to the learning rate.
  arXiv  Detail & Related papers  (2026-01-29T18:35:01Z)
• How to Set the Learning Rate for Large-Scale Pre-training? [73.03133634525635]
  We formalize this investigation into two distinct research paradigms: Fitting and Transfer.<n>Within the Fitting Paradigm, we introduce a Scaling Law for search factor, effectively reducing the search complexity from O(n3) to O(n*C_D*C_) via predictive modeling.<n>We extend the principles of $$Transfer to the Mixture of Experts (MoE) architecture, broadening its applicability to encompass model depth, weight decay, and token horizons.
  arXiv  Detail & Related papers  (2026-01-08T15:55:13Z)
• A Convergence Analysis of Adaptive Optimizers under Floating-point Quantization [32.97211471008323]
  We introduce the first theoretical framework of adaptive convergences, including Adam and Muon, under floating-point quantization of gradients, weights, and states.<n>We show that both algorithms retain convergence rates close to their full-precision counterparts provided mantissa length scales only logarithmically with the number of iterations.<n>Our analysis further reveals that Adam is highly sensitive to and second-moment quantization weights due to its reliance on $beta to 1$, while Muon requires weaker error control and is thus potentially more robust.
  arXiv  Detail & Related papers  (2025-10-24T10:16:23Z)
• Beyond Outliers: A Study of Optimizers Under Quantization [82.75879062804955]
  We study impact of choice on model robustness under quantization.<n>We evaluate how model performance degrades when trained with different baselines.<n>We derive scaling laws for quantization-aware training under different parameters.
  arXiv  Detail & Related papers  (2025-09-27T21:15:22Z)
• Efficient Edge LLMs Deployment via HessianAware Quantization and CPU GPU Collaborative [31.74122603714625]
  Mixture of Experts (MoE) architecture enhances model capacity through sparse activation.<n>MoE faces two major difficulties in practical deployment.<n>Under limited memory, efficient offloading and collaborative inference of expert modules struggle to balance latency and throughput.<n>This paper proposes an efficient MoE edge deployment scheme based on Hessian-Aware Quantization (HAQ) and CPU- GPU collaborative inference.
  arXiv  Detail & Related papers  (2025-08-10T12:59:57Z)
• QPART: Adaptive Model Quantization and Dynamic Workload Balancing for Accuracy-aware Edge Inference [10.55165549089585]
  We argue that planning an inference pattern with a request-specific model tailored to the device's computational capacity is more cost-efficient and robust to diverse scenarios.<n>We propose an accuracy-aware and workload-balanced inference system that integrates joint model quantization and inference partitioning.<n> Simulation results demonstrate a substantial reduction in overall time and power consumption, with payloads decreasing by over 80% and accuracy degradation kept below 1%.
  arXiv  Detail & Related papers  (2025-06-30T15:03:35Z)
• Supervised Optimism Correction: Be Confident When LLMs Are Sure [91.7459076316849]
  We establish a novel theoretical connection between supervised fine-tuning and offline reinforcement learning.<n>We show that the widely used beam search method suffers from unacceptable over-optimism.<n>We propose Supervised Optimism Correction, which introduces a simple yet effective auxiliary loss for token-level $Q$-value estimations.
  arXiv  Detail & Related papers  (2025-04-10T07:50:03Z)
• RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models [53.571195477043496]
  We propose an algorithm named Rotated Straight-Through-Estimator (RoSTE)<n>RoSTE combines quantization-aware supervised fine-tuning (QA-SFT) with an adaptive rotation strategy to reduce activation outliers.<n>Our findings reveal that the prediction error is directly proportional to the quantization error of the converged weights, which can be effectively managed through an optimized rotation configuration.
  arXiv  Detail & Related papers  (2025-02-13T06:44:33Z)
• BoA: Attention-aware Post-training Quantization without Backpropagation [11.096116957844014]
  Post-training quantization is a promising solution for deploying large language models on resource-constrained devices.<n>We introduce a novel backpropagation-free PTQ algorithm that optimize quantized weights by considering inter-layer dependencies.
  arXiv  Detail & Related papers  (2024-06-19T11:53:21Z)
• PLUM: Improving Inference Efficiency By Leveraging Repetition-Sparsity Trade-Off [2.326200609038491]
  Quantization and sparsity are key techniques that translate to repetition and sparsity within tensors at the hardware-software interface.<n>This paper introduces the concept of repetition-sparsity trade-off that helps explain computational efficiency during inference.<n>We propose PLUM, a unified co-design framework that integrates inference systems and quantization to leverage the repetition-sparsity trade-off.
  arXiv  Detail & Related papers  (2023-12-04T02:33:53Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.