SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translation

• URL: http://arxiv.org/abs/2505.16637v3
• Date: Fri, 20 Jun 2025 06:38:44 GMT
• Title: SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translation
• Abstract summary: We propose a Simple Self-Rewarding (SSR) Reinforcement Learning (RL) framework for machine translation (MT) that is reference-free, fully online, and relies solely on self-judging rewards.<n>Training with SSR using 13K monolingual examples and Qwen-2.5-7B as the backbone, our model SSR-Zero-7B outperforms existing MT-specific LLMs.<n>By augmenting SSR with external supervision from COMET, our strongest model, SSR-X-Zero-7B, achieves state-of-the-art performance in English $leftrightarrow$ Chinese translation.
• Score: 18.094515073693294
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have recently demonstrated remarkable capabilities in machine translation (MT). However, most advanced MT-specific LLMs heavily rely on external supervision signals during training, such as human-annotated reference data or trained reward models (RMs), which are often expensive to obtain and challenging to scale. To overcome this limitation, we propose a Simple Self-Rewarding (SSR) Reinforcement Learning (RL) framework for MT that is reference-free, fully online, and relies solely on self-judging rewards. Training with SSR using 13K monolingual examples and Qwen-2.5-7B as the backbone, our model SSR-Zero-7B outperforms existing MT-specific LLMs, e.g., TowerInstruct-13B and GemmaX-28-9B, as well as larger general LLMs like Qwen2.5-32B-Instruct in English $\leftrightarrow$ Chinese translation tasks from WMT23, WMT24, and Flores200 benchmarks. Furthermore, by augmenting SSR with external supervision from COMET, our strongest model, SSR-X-Zero-7B, achieves state-of-the-art performance in English $\leftrightarrow$ Chinese translation, surpassing all existing open-source models under 72B parameters and even outperforming closed-source models, e.g., GPT-4o and Gemini 1.5 Pro. Our analysis highlights the effectiveness of the self-rewarding mechanism compared to the external LLM-as-a-judge approach in MT and demonstrates its complementary benefits when combined with trained RMs. Our findings provide valuable insight into the potential of self-improving RL methods. We have publicly released our code, data and models.

Related papers

• North Small Translate: Advanced Cost-Effective Translation (Cohere CAT+) [51.77806201331947]
  North Small Translate is an open-weight, LLM-based machine translation (MT) model with instruction-following capabilities.<n>North Small Translate is trained using difficulty sampling to obtain challenging documents.<n>The model achieves top MT performance across 50 languages in the class of models under 1T parameters.
  arXiv  Detail & Related papers  (2026-09-12T13:02:13Z)
• RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deployment [57.588738943463646]
  Large Language Models (LLMs) have achieved remarkable performance in Machine Translation (MT)<n> deploying them at scale remains prohibitively expensive.<n>We propose textbfRouteLMT, an in-model router that predicts the expected gain by probing the small translators prompt-token representation.
  arXiv  Detail & Related papers  (2026-04-24T13:02:45Z)
• MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization [74.04867639197445]
  MiroMind-M1 is a set of fully open-source RLMs built on the Qwen-2.5-based benchmarks.<n>Our models are trained in two stages: SFT on a carefully curated corpus of 719K math-reasoning problems with verified CoT trajectories, followed by RLVR on 62K challenging and verifiable problems.
  arXiv  Detail & Related papers  (2025-07-19T16:21:23Z)
• KIT's Low-resource Speech Translation Systems for IWSLT2025: System Enhancement with Synthetic Data and Model Regularization [57.08591486199925]
  This paper presents KIT's submissions to the IWSLT 2025 low-resource track.<n>We develop both cascaded systems, and end-to-end (E2E) Speech Translation systems.<n>Building upon pre-trained models, we fine-tune our systems with different strategies to utilize resources efficiently.
  arXiv  Detail & Related papers  (2025-05-26T08:38:02Z)
• ExTrans: Multilingual Deep Reasoning Translation via Exemplar-Enhanced Reinforcement Learning [77.41383117199227]
  We design a new reward modeling method that compares the translation results of the policy MT model with a strong LRM.<n>Using Qwen2.5-7B-Instruct as the backbone, the trained model achieves the new state-of-the-art performance in literary translation.<n>We extend our method to the multilingual settings with 11 languages.
  arXiv  Detail & Related papers  (2025-05-19T11:34:47Z)
• MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learning [24.59797320737557]
  We introduce MT-R1-Zero, the first open-source adaptation of the R1-Zero RL framework for machine translation (MT)<n>On the WMT 24 English-Chinese benchmark, our MT-R1-Zero-3B-Mix achieves competitive performance, surpassing TowerInstruct-7B-v0.2 by an average of 1.26 points.<n>Our work exhibits strong generalization capabilities on out-of-distribution MT tasks, robustly supporting multilingual and low-resource settings.
  arXiv  Detail & Related papers  (2025-04-14T12:14:18Z)
• Learning to Reason via Self-Iterative Process Feedback for Small Language Models [5.3831551965806534]
  Small language models (SLMs) are more efficient, cost-effective, and customizable than large language models (LLMs)<n>This study enables SLMs to learn to reason from self-iterative feedback.
  arXiv  Detail & Related papers  (2024-12-11T14:05:04Z)
• Reward Guidance for Reinforcement Learning Tasks Based on Large Language Models: The LMGT Framework [1.5802986215292307]
  Language Model Guided reward Tuning (LMGT) is a novel, sample-efficient framework for Reinforcement Learning.<n>We show that LMGT adeptly balances exploration and exploitation, thereby guiding the agent's exploratory behavior.<n>Our results suggest that LMGT can substantially reduce the computational resources required during the RL training phase.
  arXiv  Detail & Related papers  (2024-09-07T07:40:43Z)
• Blending LLMs into Cascaded Speech Translation: KIT's Offline Speech Translation System for IWSLT 2024 [61.189875635090225]
  Large Language Models (LLMs) are currently under exploration for various tasks, including Automatic Speech Recognition (ASR), Machine Translation (MT), and even End-to-End Speech Translation (ST)
  arXiv  Detail & Related papers  (2024-06-24T16:38:17Z)
• Ladder: A Model-Agnostic Framework Boosting LLM-based Machine Translation to the Next Level [9.699022347910121]
  General-purpose Large Language Models (LLMs) have achieved remarkable advancements in machine translation (MT) by leveraging extensive web content. However, translation-specific LLMs are built by pre-training on domain-specific monolingual corpora and fine-tuning with human-annotated translation data. We develop MT-Ladder, a novel model-agnostic and cost-effective tool to refine the performance of general LLMs for MT.
  arXiv  Detail & Related papers  (2024-06-22T05:33:35Z)
• Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation [50.00235162432848]
  We train ALMA models with only 22K parallel sentences and 12M parameters. The resulting model, called ALMA-R, can match or exceed the performance of the WMT competition winners and GPT-4.
  arXiv  Detail & Related papers  (2024-01-16T15:04:51Z)
• The USYD-JD Speech Translation System for IWSLT 2021 [85.64797317290349]
  This paper describes the University of Sydney& JD's joint submission of the IWSLT 2021 low resource speech translation task. We trained our models with the officially provided ASR and MT datasets. To achieve better translation performance, we explored the most recent effective strategies, including back translation, knowledge distillation, multi-feature reranking and transductive finetuning.
  arXiv  Detail & Related papers  (2021-07-24T09:53:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.