Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax

• URL: http://arxiv.org/abs/2605.14366v1
• Date: Thu, 14 May 2026 04:47:22 GMT
• Title: Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax
• Abstract summary: We propose a semantic-space alignment paradigm powered by Group Relative Policy (GRPO)<n>This objective encourages meaning through flexible realizations, enabling controlled updates that reduce destructive interference with pretrained knowledge.<n>Experiments show that our method acquires low-resource capabilities while markedly mitigating alignment tax.
• Score: 17.596121359044847
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Extending large language models (LLMs) to low-resource languages often incurs an "alignment tax": improvements in the target language come at the cost of catastrophic forgetting in general capabilities. We argue that this trade-off arises from the rigidity of supervised fine-tuning (SFT), which enforces token-level surface imitation on narrow and biased data distributions. To address this limitation, we propose a semantic-space alignment paradigm powered by Group Relative Policy Optimization (GRPO), where the model is optimized using embedding-level semantic rewards rather than likelihood maximization. This objective encourages meaning preservation through flexible realizations, enabling controlled updates that reduce destructive interference with pretrained knowledge. We evaluate our approach on Tibetan-Chinese machine translation and Tibetan headline generation. Experiments show that our method acquires low-resource capabilities while markedly mitigating alignment tax, preserving general competence more effectively than SFT. Despite producing less rigid surface overlap, semantic RL yields higher semantic quality and preference in open-ended generation, and few-shot transfer results indicate that it learns more transferable and robust representations under limited supervision. Overall, our study demonstrates that reinforcement learning with semantic rewards provides a safer and more reliable pathway for inclusive low-resource language expansion.

Related papers

• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• Unified Gradient Projection: Language-Balanced Continual Learning for Multilingual Low-Resource ASR [6.818862582399845]
  Fine-tuning on low-resource languages often causes catastrophic forgetting.<n>We propose Unified Gradient Projection, which constrains parameter updates using reference gradients from language-balanced replay.<n>On Whisper-large-v3, it achieves near-zero average forgetting.
  arXiv  Detail & Related papers  (2026-07-13T06:57:02Z)
• Source-Grounded Semantic Reinforcement Learning for Low-Resource Target-Language Generation [19.750295463954853]
  Source-Grounded Semantic Reinforcement Learning (SG-SRL) is a resource-utilization framework that converts source-language monolingual data into cross-lingual semantic supervision for target-language generation.<n>SG-SRL performs reference-free reinforcement learning (RL) on source-language data using a cross-lingual semantic reward model.<n>Experiments on Chinese-to-Thai generation show that SG-SRL improves semantic grounding and factual coverage over cold-start SFT.
  arXiv  Detail & Related papers  (2026-05-28T07:27:16Z)
• STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning [80.78140312980484]
  We propose language-driven stepwise trajectory redirection, termed as STRIDE.<n>We co-train a generator and a generative verifier using only outcome-based rewards, eliminating external annotations.<n> Experiments on diverse reasoning benchmarks show that STRIDE significantly outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-05-13T11:04:31Z)
• Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models [65.89572755202245]
  Diffusion Language Models (DLMs) offer stronger global awareness and highly parallel generation.<n>Post-training DLMs with standard Negative Evidence Lower Bound (NELBO)-based supervised fine-tuning remains inefficient.<n>We propose a self-distilled trajectory-based post-training framework that aligns training with the easy-to-hard structure of inference.
  arXiv  Detail & Related papers  (2026-05-12T09:39:06Z)
• MERIT: Multilingual Expert-Reward Informed Tuning for Chinese-Centric Low-Resource Machine Translation [9.42682887916488]
  We introduce a Chinese-centric ALT benchmark for five Southeast Asian low-resource languages (LRLs)<n>In LRLtextrightChinese, fine-tuning and reward-guided optimization dramatically outperform mere model scaling.<n>These results confirm that, in LRLtextrightChinese, targeted data and reward-guided optimization dramatically outperform mere model scaling.
  arXiv  Detail & Related papers  (2026-04-06T16:42:34Z)
• Compensating Visual Insufficiency with Stratified Language Guidance for Long-Tail Class Incremental Learning [50.795452298625996]
  Long-tail class incremental learning (LT CIL) remains highly challenging because the scarcity of samples in tail classes hampers their learning.<n>We exploit the informativeness and scalability of language knowledge to tackle these issues.<n>We introduce stratified adaptive language guidance, which leverages learnable weights to merge multi-scale semantic representations.
  arXiv  Detail & Related papers  (2026-03-23T08:50:00Z)
• VEPO: Variable Entropy Policy Optimization for Low-Resource Language Foundation Models [17.595722907548062]
  Large language models exhibit suboptimal performance on low resource languages.<n>We propose Variable Entropy Policy Optimization (VEPO), which incorporates deterministic structural constraints into the policy alignment process.<n>VePO yields substantial improvements in both tokenization efficiency and translation quality, bridging the performance gap for underrepresented languages.
  arXiv  Detail & Related papers  (2026-03-19T17:10:29Z)
• LAD: Learning Advantage Distribution for Reasoning [11.179134756179998]
  We introduce Learning Advantage Distributions, a distribution-matching framework that replaces advantage with learning advantage-induced distribution.<n>We show that LAD reliably improves both accuracy and generative diversity.<n> Experiments on math and code reasoning tasks show that LAD reliably improves both accuracy and generative diversity.
  arXiv  Detail & Related papers  (2026-02-23T18:44:10Z)
• MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization [56.074760766965085]
  Group-Relative Policy Optimization has emerged as an efficient paradigm for aligning Large Language Models (LLMs)<n>We propose MAESTRO, which treats reward scalarization as a dynamic latent policy, leveraging the model's terminal hidden states as a semantic bottleneck.<n>We formulate this as a contextual bandit problem within a bi-level optimization framework, where a lightweight Conductor network co-evolves with the policy by utilizing group-relative advantages as a meta-reward signal.
  arXiv  Detail & Related papers  (2026-01-12T05:02:48Z)
• Exposing the Cracks: Vulnerabilities of Retrieval-Augmented LLM-based Machine Translation [40.37732389445079]
  We propose a noise synthesis framework and new metrics to evaluate the robustness of REAL-MT.<n>Our results show that low-resource language pairs, which rely more heavily on retrieved context, degrade more severely under noise.<n>Although LRMs possess enhanced reasoning capabilities, they show no improvement in error correction and are even more susceptible to noise.
  arXiv  Detail & Related papers  (2025-10-01T12:43:55Z)
• Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections [65.36449542323277]
  We present a unified theoretical framework bridgingSupervised Fine-Tuning (SFT) and preference learning in Large Language Model (LLM) post-training.<n>We propose a simple yet effective learning rate reduction approach that yields significant performance improvements.
  arXiv  Detail & Related papers  (2025-06-15T05:42:29Z)
• TRACE: TRansformer-based Attribution using Contrastive Embeddings in LLMs [50.259001311894295]
  We propose a novel TRansformer-based Attribution framework using Contrastive Embeddings called TRACE. We show that TRACE significantly improves the ability to attribute sources accurately, making it a valuable tool for enhancing the reliability and trustworthiness of large language models.
  arXiv  Detail & Related papers  (2024-07-06T07:19:30Z)
• Safe Multi-agent Reinforcement Learning with Natural Language Constraints [49.01100552946231]
  The role of natural language constraints in Safe Multi-agent Reinforcement Learning (MARL) is crucial, yet often overlooked. We propose a novel approach named Safe Multi-agent Reinforcement Learning with Natural Language constraints (SMALL) Our method leverages fine-tuned language models to interpret and process free-form textual constraints, converting them into semantic embeddings. These embeddings are then integrated into the multi-agent policy learning process, enabling agents to learn policies that minimize constraint violations while optimizing rewards.
  arXiv  Detail & Related papers  (2024-05-30T12:57:35Z)
• Semi-supervised Neural Machine Translation with Consistency Regularization for Low-Resource Languages [3.475371300689165]
  This paper presents a simple yet effective method to tackle the problem for low-resource languages by augmenting high-quality sentence pairs and training NMT models in a semi-supervised manner. Specifically, our approach combines the cross-entropy loss for supervised learning with KL Divergence for unsupervised fashion given pseudo and augmented target sentences. Experimental results show that our approach significantly improves NMT baselines, especially on low-resource datasets with 0.46--2.03 BLEU scores.
  arXiv  Detail & Related papers  (2023-04-02T15:24:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.