Exposing the Cracks: Vulnerabilities of Retrieval-Augmented LLM-based Machine Translation

• URL: http://arxiv.org/abs/2510.00829v1
• Date: Wed, 01 Oct 2025 12:43:55 GMT
• Title: Exposing the Cracks: Vulnerabilities of Retrieval-Augmented LLM-based Machine Translation
• Authors: Yanming Sun, Runzhe Zhan, Chi Seng Cheang, Han Wu, Xuebo Liu, Yuyao Niu, Fengying Ye, Kaixin Lan, Lidia S. Chao, Derek F. Wong,
• Abstract summary: We propose a noise synthesis framework and new metrics to evaluate the robustness of REAL-MT.<n>Our results show that low-resource language pairs, which rely more heavily on retrieved context, degrade more severely under noise.<n>Although LRMs possess enhanced reasoning capabilities, they show no improvement in error correction and are even more susceptible to noise.
• Score: 40.37732389445079
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: \textbf{RE}trieval-\textbf{A}ugmented \textbf{L}LM-based \textbf{M}achine \textbf{T}ranslation (REAL-MT) shows promise for knowledge-intensive tasks like idiomatic translation, but its reliability under noisy retrieval contexts remains poorly understood despite this being a common challenge in real-world deployment. To address this gap, we propose a noise synthesis framework and new metrics to evaluate the robustness of REAL-MT systematically. Using this framework, we instantiate REAL-MT with Qwen-series models, including standard LLMs and large reasoning models (LRMs) with enhanced reasoning, and evaluate their performance on idiomatic translation across high-, medium-, and low-resource language pairs under synthesized noise. Our results show that low-resource language pairs, which rely more heavily on retrieved context, degrade more severely under noise than high-resource ones and often produce nonsensical translations. Although LRMs possess enhanced reasoning capabilities, they show no improvement in error correction and are even more susceptible to noise, tending to rationalize incorrect contexts. We find that this stems from an attention shift away from the source idiom to noisy content, while confidence increases despite declining accuracy, indicating poor calibration. To mitigate these issues, we investigate training-free and fine-tuning strategies, which improve robustness at the cost of performance in clean contexts, revealing a fundamental trade-off. Our findings highlight the limitations of current approaches, underscoring the need for self-verifying integration mechanisms.

Related papers

• Unlocking Reasoning Capability on Machine Translation in Large Language Models [57.60641851466707]
  Reasoning-oriented large language models (RLMs) achieve strong gains on tasks such as mathematics and coding by generating explicit intermediate reasoning.<n>We systematically evaluate several open- and closed-weights RLMs on the WMT24++ benchmark.<n>We find that enabling explicit reasoning consistently degrades translation quality across languages and models.
  arXiv  Detail & Related papers  (2026-02-16T14:05:59Z)
• Unlocking the Potentials of Retrieval-Augmented Generation for Diffusion Language Models [38.148737920360766]
  Retrieval-Augmented Generation shows great successes for enhancing large language models (LLMs)<n>We show that DLMs coupled with RAG show promising potentials with stronger dependency on contextual information, but suffer from limited generation precision.<n>We propose Semantic-Preserving REtrieval-Augmented Diffusion (SPREAD), a novel framework that introduces a query-relevance-guided denoising strategy.
  arXiv  Detail & Related papers  (2026-01-16T14:45:46Z)
• NAACL: Noise-AwAre Verbal Confidence Calibration for LLMs in RAG Systems [53.52419750390942]
  Large language models (LLMs) are used in mission-critical factual domains.<n>LLMs exhibit poor calibration performance due to noisy retrieved contexts.<n>We propose NAACL Rules (Noise-AwAre Confidence CaLibration Rules) to provide a principled foundation for resolving overconfidence under noise.
  arXiv  Detail & Related papers  (2026-01-16T05:38:25Z)
• Lost in the Noise: How Reasoning Models Fail with Contextual Distractors [57.31788955167306]
  Recent advances in reasoning models and agentic AI systems have led to an increased reliance on diverse external information.<n>We introduce NoisyBench, a comprehensive benchmark that systematically evaluates model robustness across 11 datasets in RAG, reasoning, alignment, and tool-use tasks.<n>Our evaluation reveals a catastrophic performance drop of up to 80% in state-of-the-art models when faced with contextual distractors.
  arXiv  Detail & Related papers  (2026-01-12T05:43:51Z)
• Rethinking LLM Parametric Knowledge as Post-retrieval Confidence for Dynamic Retrieval and Reranking [23.1400319714807]
  Large Language Models (LLMs) often generate inaccurate responses (hallucinations) when faced with questions beyond their knowledge scope.<n>Retrieval-Augmented Generation (RAG) addresses this by leveraging external knowledge, but a critical challenge remains: determining whether retrieved contexts effectively enhance the models ability to answer specific queries.<n>This challenge underscores the importance of knowledge boundary awareness, which current methods-relying on discrete labels or limited signals-fail to address adequately.
  arXiv  Detail & Related papers  (2025-09-08T09:37:20Z)
• Context Engineering for Trustworthiness: Rescorla Wagner Steering Under Mixed and Inappropriate Contexts [55.70338710797578]
  We introduce the Poisoned Context Testbed, pairing queries with real-world contexts containing relevant and inappropriate content.<n>Inspired by associative learning in animals, we adapt the Rescorla-Wagner (RW) model from neuroscience to quantify how competing contextual signals influence LLM outputs.<n>We introduce RW-Steering, a two-stage finetuning-based approach that enables the model to internally identify and ignore inappropriate signals.
  arXiv  Detail & Related papers  (2025-09-02T00:40:34Z)
• From Entity Reliability to Clean Feedback: An Entity-Aware Denoising Framework Beyond Interaction-Level Signals [20.323837731778358]
  Implicit feedback is central to recommender systems but is inherently noisy, often impairing model training and degrading user experience.<n>We propose textbfEARD (textbfEntity-textbfAware textbfReliability-textbfDriven Denoising), a lightweight framework that shifts the focus from interaction-level signals to entity-level reliability.
  arXiv  Detail & Related papers  (2025-08-14T17:20:12Z)
• Exploring LLM Reasoning Through Controlled Prompt Variations [0.9217021281095907]
  We evaluate how well state-of-the-art models maintain logical consistency and correctness when confronted with four categories of prompt perturbations.<n>Our experiments, conducted on thirteen open-source and closed-source LLMs, reveal that introducing irrelevant context within the model's context window significantly degrades performance.<n>Certain perturbations inadvertently trigger chain-of-thought-like reasoning behaviors, even without explicit prompting.
  arXiv  Detail & Related papers  (2025-04-02T20:18:50Z)
• Refining Translations with LLMs: A Constraint-Aware Iterative Prompting Approach [7.5069214839655345]
  Large language models (LLMs) have demonstrated remarkable proficiency in machine translation (MT) We propose a multi-step prompt chain that enhances translation faithfulness by prioritizing key terms crucial for semantic accuracy. Experiments using Llama and Qwen as base models on the FLORES-200 and WMT datasets demonstrate significant improvements over baselines.
  arXiv  Detail & Related papers  (2024-11-13T05:40:24Z)
• SMILE: Speech Meta In-Context Learning for Low-Resource Language Automatic Speech Recognition [55.2480439325792]
  Speech Meta In-Context LEarning (SMILE) is an innovative framework that combines meta-learning with speech in-context learning (SICL)<n>We show that SMILE consistently outperforms baseline methods in training-free few-shot multilingual ASR tasks.
  arXiv  Detail & Related papers  (2024-09-16T16:04:16Z)
• Disentangled Noisy Correspondence Learning [56.06801962154915]
  Cross-modal retrieval is crucial in understanding latent correspondences across modalities. DisNCL is a novel information-theoretic framework for feature Disentanglement in Noisy Correspondence Learning.
  arXiv  Detail & Related papers  (2024-08-10T09:49:55Z)
• Large Language Models are Efficient Learners of Noise-Robust Speech Recognition [65.95847272465124]
  Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR) In this work, we extend the benchmark to noisy conditions and investigate if we can teach LLMs to perform denoising for GER. Experiments on various latest LLMs demonstrate our approach achieves a new breakthrough with up to 53.9% correction improvement in terms of word error rate.
  arXiv  Detail & Related papers  (2024-01-19T01:29:27Z)
• Joint Dropout: Improving Generalizability in Low-Resource Neural Machine Translation through Phrase Pair Variables [17.300004156754966]
  We propose a method called Joint Dropout, that addresses the challenge of low-resource neural machine translation by substituting phrases with variables. We observe a substantial improvement in translation quality for language pairs with minimal resources, as seen in BLEU and Direct Assessment scores.
  arXiv  Detail & Related papers  (2023-07-24T14:33:49Z)
• Enhancing Contrastive Learning with Noise-Guided Attack: Towards Continual Relation Extraction in the Wild [57.468184469589744]
  We develop a noise-resistant contrastive framework named as textbfNoise-guided textbfattack in textbfContrative textbfLearning(NaCL) Compared to direct noise discarding or inaccessible noise relabeling, we present modifying the feature space to match the given noisy labels via attacking.
  arXiv  Detail & Related papers  (2023-05-11T18:48:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.