Studying quantization trade-offs for efficient inference deployment in machine translation

• URL: http://arxiv.org/abs/2607.29397v2
• Date: Tue, 04 Aug 2026 16:14:13 GMT
• Title: Studying quantization trade-offs for efficient inference deployment in machine translation
• Abstract summary: We study the quantization trade-offs of two translation model families, EuroLLM citepmartins2025eurollm and Hy-MT2 citepzheng2026hy across five models for efficient deployment on a single A100 or H100 GPU.
• Score: 6.265237910291255
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Deploying large language models in realistic server environments poses challenges, as the system needs to provide high-quality responses with low latency. Quantization is a common approach to reduce the memory footprint and improve inference efficiency, yet its impact on latency and throughput is rarely evaluated under controlled, orchestration-level workloads. In this work we study the quantization trade-offs of two translation model families, EuroLLM \citep{martins2025eurollm} and Hy-MT2 \citep{zheng2026hy} across five models ranging from 1.7B to 22B for efficient deployment on a single A100 or H100 GPU. We demonstrate that combining a document-chunking strategy with W4A8 or W8A8 quantization improves the latency-throughput Pareto-curve under a wide range of workloads. Furthermore, since standard machine translation (MT) benchmarks rely on isolated sentences and fail to capture long-context dynamics, we introduce a document-level evaluation from WMT24++ to assess how text chunking strategies affect translation quality under quantization. Our results reveal that standard segment-level evaluation can fail to predict the interaction between quantization and long-context document translation. While Hy-MT2 remains robust under quantization, EuroLLM shows strong sensitivity and translation quality collapses rapidly for all considered quantization formats. Overall, our experiments show that the trade-off between inference efficiency and translation quality depends not only on the quantization format, but also on the choice of text chunking strategy.

Related papers

• Generating Adversarial Texts for Machine Translation via GRPO [9.016860740260679]
  We propose a scalable reinforcement-learning-based approach for rewriting existing source texts.<n>On WMT25, our approach substantially reduces average COMET translation quality from 0.63 to 0.48, while preserving grammaticality and readability.<n> Evaluations on the unseen WMT19-WMT24 benchmarks confirm that this behavior generalizes beyond the training data.
  arXiv  Detail & Related papers  (2026-09-05T12:06:59Z)
• Hierarchical Policy Optimization for Simultaneous Translation of Unbounded Speech [50.45710815530982]
  Simultaneous speech translation (SST) generates translations while receiving partial speech input.<n>Recent advances show that large language models (LLMs) can substantially improve SST quality, but at the cost of high computational overhead.<n>We propose a Hierarchical Policy Optimization (HPO) approach that trains post-train models on imperfect SFT data.<n> Experiments on English to Chinese/German/Japanese demonstrate improvements of over +7 COMET score and +1.25 MetricX score at a latency of 1.5 seconds.
  arXiv  Detail & Related papers  (2026-04-22T19:43:51Z)
• ReflectMT: Internalizing Reflection for Efficient and High-Quality Machine Translation [63.89730738735446]
  We propose ReflectMT, a two-stage reflection internalization algorithm for machine translation.<n>Our approach develops the model's "translate-reflect-refine" capability through reinforcement learning.<n> Experimental results on datasets such as WMT24 demonstrate that our model's first-pass translations during inference outperform multi-step reasoning LRMs.
  arXiv  Detail & Related papers  (2026-04-21T06:48:41Z)
• Incentivizing Parametric Knowledge via Reinforcement Learning with Verifiable Rewards for Cross-Cultural Entity Translation [68.85147984815778]
  We propose EA-RLVR (Entity-Anchored Reinforcement Learning with Verifiable Rewards) to incentivize the effective use of parametric knowledge.<n>EA-RLVR anchors supervision on a verifiable, entity-level reward signal and incorporates lightweight structural gates to stabilize optimization.<n>We evaluate EA-RLVR on XC-Translate and observe consistent improvements in both entity translation accuracy and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-04-18T07:15:43Z)
• The Uneven Impact of Post-Training Quantization in Machine Translation [6.398727997282354]
  Quantization is essential for deploying large language models (LLMs) on resource-constrained hardware, but its implications for multilingual tasks remain underexplored.<n>We conduct the first large-scale evaluation of post-training quantization (PTQ) on machine translation across 55 languages using five LLMs ranging from 1.7B to 70B parameters.<n>Our analysis reveals that while 4-bit quantization often preserves translation quality for high-resource languages, significant degradation occurs for low-resource and typologically diverse languages, particularly in 2-bit settings.
  arXiv  Detail & Related papers  (2025-08-28T15:22:31Z)
• "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization [67.3213104337679]
  Quantization is a powerful tool for accelerating large language model (LLM) inference, but the accuracy-performance trade-offs across different formats remain unclear.<n>We conduct the most comprehensive empirical study to date, evaluating FP8, INT8, and INT4 quantization across academic benchmarks and real-world tasks.
  arXiv  Detail & Related papers  (2024-11-04T18:21:59Z)
• Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models [57.80514758695275]
  Using large language models (LLMs) for assessing the quality of machine translation (MT) achieves state-of-the-art performance at the system level. We propose a new prompting method called textbftextttError Analysis Prompting (EAPrompt) This technique emulates the commonly accepted human evaluation framework - Multidimensional Quality Metrics (MQM) and textitproduces explainable and reliable MT evaluations at both the system and segment level.
  arXiv  Detail & Related papers  (2023-03-24T05:05:03Z)
• Measuring Uncertainty in Translation Quality Evaluation (TQE) [62.997667081978825]
  This work carries out motivated research to correctly estimate the confidence intervals citeBrown_etal2001Interval depending on the sample size of the translated text. The methodology we applied for this work is from Bernoulli Statistical Distribution Modelling (BSDM) and Monte Carlo Sampling Analysis (MCSA)
  arXiv  Detail & Related papers  (2021-11-15T12:09:08Z)
• Improving Multilingual Translation by Representation and Gradient Regularization [82.42760103045083]
  We propose a joint approach to regularize NMT models at both representation-level and gradient-level. Our results demonstrate that our approach is highly effective in both reducing off-target translation occurrences and improving zero-shot translation performance.
  arXiv  Detail & Related papers  (2021-09-10T10:52:21Z)
• Verdi: Quality Estimation and Error Detection for Bilingual [23.485380293716272]
  Verdi is a novel framework for word-level and sentence-level post-editing effort estimation for bilingual corpora. We exploit the symmetric nature of bilingual corpora and apply model-level dual learning in the NMT predictor. Our method beats the winner of the competition and outperforms other baseline methods by a great margin.
  arXiv  Detail & Related papers  (2021-05-31T11:04:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.