Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs

• URL: http://arxiv.org/abs/2607.19243v1
• Date: Tue, 21 Jul 2026 16:15:05 GMT
• Title: Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs
• Abstract summary: Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
• Score: 51.56484100374058
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although Large Language Models (LLMs) demonstrate remarkable multilingual fluency, their internal knowledge representations remain disproportionately biased toward high-resource languages. This leads to cross-lingual factual inconsistency, where they shift their empirical answer distributions based solely on the prompt language. We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages (German, Spanish, Bulgarian), and evaluate four intervention strategies: zero-shot contextual steering (persona prompting), internal representation manipulation via Contrastive Activation Addition (CAA), and lightweight weight modification via Direct Preference Optimization (DPO) trained on benchmark-derived factual data as well as conceptual generalization data. To assess alignment, we curate a multilingual factual dataset alongside a novel generalization benchmark comprising culturally rooted queries to determine whether factual interventions transfer to broader target-centric preferences. Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization. While CAA yields sharp inconsistency benchmark shifts, it is configuration-sensitive and risks knowledge degradation. DPO-based adapters offer permanent, yet narrower and less transferable gains. These findings suggest that cross-lingual inconsistency is at least partly a selection problem, and that simple contextual interventions may outperform more invasive methods for robust, transferable alignment.

Related papers

• STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning [80.78140312980484]
  We propose language-driven stepwise trajectory redirection, termed as STRIDE.<n>We co-train a generator and a generative verifier using only outcome-based rewards, eliminating external annotations.<n> Experiments on diverse reasoning benchmarks show that STRIDE significantly outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-05-13T11:04:31Z)
• Multilingual Refusal Alignment for Safer Large Language Models [53.64286756804503]
  We investigate whether single-language alignment transfers cross-lingually, how language consistency is preserved during training, and the resulting trade-offs with general knowledge capabilities.<n>We introduce RefusEU, a novel refusal alignment dataset covering 12 European languages, including a dedicated test set for evaluating current state-of-the-art models.<n>Our controlled Direct Preference Optimization (DPO) experiments provide two key insights: aligning models exclusively in English is insufficient to ensure cross-lingual safety, even for the same harm categories, whereas training on multilingual datasets can improve safety without degrading general performance, as measured by the Global MMLU benchmark.
  arXiv  Detail & Related papers  (2026-04-24T09:29:14Z)
• Mitigating Cross-Lingual Cultural Inconsistencies in LLMs via Consensus-Driven Preference Optimisation [58.01855677487771]
  We show that multilingual large language models (MLLMs) frequently exhibit inconsistent behaviour when the prompt's language changes.<n>We propose Cross-lingual Cultural Consistent Preference optimisation (C-3PO), a consensus-driven alignment framework.<n>C-3PO achieves up to a 0.10-point absolute increase in $_S$ over unaligned models, outperforming strong prompting and representation steering baselines.
  arXiv  Detail & Related papers  (2026-04-02T14:04:06Z)
• Translation or Recitation? Calibrating Evaluation Scores for Machine Translation of Extremely Low-Resource Languages [39.985923582735936]
  We introduce the FRED Difficulty Metrics, which include the Fertility Ratio (F), Retrieval Proxy (R), Pre-training Exposure (E) and Corpus Diversity (D)<n>These metrics reveal that a significant portion of result variability is explained by train-test overlap and pre-training exposure rather than model capability.
  arXiv  Detail & Related papers  (2026-03-26T09:20:17Z)
• Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck [37.780081880731096]
  Large language models (LLMs) have become a standard for multilingual evaluation, yet they exhibit a severe systematic translationese bias.<n>We attribute this bias to spurious correlations with (i) latent manifold alignment with English and (ii) cross-lingual predictability.<n>To mitigate this bias, we propose DIBJudge, a robust fine-tuning framework that learns a minimally sufficient, judgment-critical representation.
  arXiv  Detail & Related papers  (2026-03-11T02:55:29Z)
• BiasLab: A Multilingual, Dual-Framing Framework for Robust Measurement of Output-Level Bias in Large Language Models [3.643198597030366]
  This paper introduces BiasLab, an open-source, model-agnostic evaluation framework for quantifying output-level (extrinsic) bias.<n>The framework supports evaluation across diverse bias axes, including demographic, cultural, political, and geopolitical topics.
  arXiv  Detail & Related papers  (2026-01-11T11:07:46Z)
• Parallel Universes, Parallel Languages: A Comprehensive Study on LLM-based Multilingual Counterfactual Example Generation [49.2073409243885]
  Large language models (LLMs) excel at generating English counterfactuals and demonstrate multilingual proficiency.<n>We conduct automatic evaluations on both directly generated counterfactuals in the target languages and those derived via English translation across six languages.<n>We identify and categorize four main types of errors that consistently appear in the generated counterfactuals across languages.
  arXiv  Detail & Related papers  (2026-01-01T08:53:49Z)
• LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs [73.27182315028021]
  LANPO is a framework that cleanly separates the roles of feedback: language guides exploration, while numerical rewards drive optimization.<n>Our work provides a robust method for integrating historical experiences into the LLM RL loop, creating more effective and data-efficient learning agents.
  arXiv  Detail & Related papers  (2025-10-18T15:51:19Z)
• CogBench: A Large Language Model Benchmark for Multilingual Speech-Based Cognitive Impairment Assessment [23.1730341293796]
  We propose CogBench, the first benchmark designed to evaluate the cross-lingual and cross-site generalizability of large language models for speech-based cognitive impairment assessment.<n>Our results show that conventional deep learning models degrade substantially when transferred across domains.<n>Our findings offer a critical step toward building clinically useful and linguistically robust speech-based cognitive assessment tools.
  arXiv  Detail & Related papers  (2025-08-05T12:06:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.