Interpretable Cross-Lingual Alignment in Small Language Models: Probing Cultural and Pragmatic Reasoning in Japanese-English Bilingual LLMs

• URL: http://arxiv.org/abs/2608.14896v1
• Date: Fri, 14 Aug 2026 21:04:21 GMT
• Title: Interpretable Cross-Lingual Alignment in Small Language Models: Probing Cultural and Pragmatic Reasoning in Japanese-English Bilingual LLMs
• Abstract summary: I introduce J-PragEval-v0, a minimal-pair benchmark isolating four phenomena from surface fluency.<n>I combine it with linear probes and teacher-forced log-probability evaluation to ask where inside TinySwallow-1.5B the corresponding contrasts live.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models work well on English and behave in poorly understood ways on languages typologically far from it. Japanese is a clean example, where evaluation still leans on translation quality and JGLUE-style benchmarks, which roll lexical, syntactic and pragmatic competence into a single score. The phenomena on which general-purpose models fail Japanese users are pragmatic: honorifics, in-group and out-group reference, context-sensitive politeness, zero anaphora. I introduce J-PragEval-v0, a minimal-pair benchmark isolating four such phenomena from surface fluency, and combine it with linear probes and teacher-forced log-probability evaluation to ask where inside TinySwallow-1.5B (28 layers, hidden size 1536) the corresponding contrasts live. The four features split three ways. Honorific register sits cleanly in the residual stream: 0.96 balanced accuracy at layer 15, and the model flips its preferred continuation with the scenario on 93 percent of items. Implicit subject and in-group reference are not linearly decodable at the final prompt token (0.48 and 0.38), yet flip rates are 0.77 and 0.79, so the contrast is worked out during generation rather than stored at the prompt. Indirect refusal is the negative case: 0.95 probe accuracy collapsing to a 0.43 flip rate under length-normalised teacher forcing, because the current minimal pairs conflate politeness with continuation length. I also specify Pragmatic Representation Steering, a parameter-free inference-time method that edits residual-stream activations along the class-mean-difference directions probing identifies. Feasibility is argued indirectly rather than demonstrated: the contrastive activation addition baseline, the same geometry the method would inject, recovers probe accuracy within one to two points of logistic regression wherever a linear signal exists. Scaling to Llama-3.1-Swallow-8B is the next step.

Related papers

• Beyond the Name: Demographic Leakage in De-Identified Résumés and Evaluation Artifacts in LLM Bias Audits [4.736836688901646]
  We investigate whether eliminating language fields resolves leakage across nine open-weight models and 620 résumés.<n>We isolate unstructured prose across five ethnocultural conditions and three cue-salience tiers.<n>Target-group recovery averages 0.757 overall and saturates at 1.000 under high salience.
  arXiv  Detail & Related papers  (2026-09-15T01:48:44Z)
• When Do Internal Probes Beat Reading the Answer? Miscalibrated Readouts and Behavior-Concealed Knowledge in Language Models [0.0]
  A language model, asked to verify 1,200 logical conclusions, answers YES every time.<n>linear probes on its hidden states read the correct verdict at 0.96 AUC.<n>We ask where the verdict is lost, and find the dominant failure is a single scalar.<n>The diagnosis generalizes: across 90 semantic-label configurations of a five-model, three-family factorial, behavioral accuracy collapses onto a single function of threshold offset.
  arXiv  Detail & Related papers  (2026-09-04T00:26:45Z)
• Structure, Association, and Decision Value: Representation-Based Difficulty Estimation for Adaptive Inference in African-Language NLI [0.0]
  We study natural language inference across 15 African languages with frozen off-the-shelf checkpoints.<n>Because AfriXNLI is derived from XNLI, its English, French and Swahili configurations cannot serve as clean evaluations for XNLI-trained models.<n>Our central methodological finding is that a representation statistic can be statistically significant for one notion of computational benefit while being irrelevant to another.
  arXiv  Detail & Related papers  (2026-08-19T15:03:51Z)
• TokenPrint: A Calibrated Token-Space Fingerprint for Language-Model Provenance [8.657685524975246]
  We introduce a training-free fingerprint based on the top-$k$ vocabulary projections of late hidden states elicited by 250 fixed knowledge probes.<n>We evaluate the method on 32 open-weight models from nine families (0.6B--32B) with documented relationships.
  arXiv  Detail & Related papers  (2026-08-08T13:56:39Z)
• OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment [0.9821874476902969]
  We introduce OptimismBench, which detects directional bias with inverted pairs.<n>We show post-training sets the sign of the bias, with opposite shifts in different families.<n>We release 3,870 items across 10 languages for per-model directional-bias auditing.
  arXiv  Detail & Related papers  (2026-07-29T14:38:55Z)
• Multilingual Embedding Probes Fail to Generalize Across Learner Corpora [0.17188280334580197]
  We train probes on hidden-state activations to predict CEFR proficiency levels from learner texts across nine corpora and seven languages.<n>Under in-distribution evaluation, probes achieve strong performance, substantially outperforming the surface baseline.<n>In cross-corpus evaluation performance collapses across all probe types and model sizes.<n>These results suggest that current multilingual embeddings do not straightforwardly encode language-general proficiency.
  arXiv  Detail & Related papers  (2026-04-08T13:47:54Z)
• Bilingual Text-to-Motion Generation: A New Benchmark and Baselines [52.71312720094036]
  We introduce BiHumanML3D, the first bilingual text-to-motion benchmark, constructed via LLM-assisted annotation and rigorous manual correction.<n>We also propose a simple yet effective baseline, Bilingual Motion Diffusion synthesis (BiMD), featuring Cross-Lingual Alignment (CLA)<n>BiMD with CLA achieves an FID of 0.045 vs. 0.169 and R@3 of 82.8% vs. 80.8%, significantly outperforms monolingual diffusion models and translation baselines on BiHumanML3D.
  arXiv  Detail & Related papers  (2026-03-26T08:48:27Z)
• Cross-Lingual Stability of LLM Judges Under Controlled Generation: Evidence from Finno-Ugric Languages [0.22009842278462158]
  Cross-lingual evaluation of large language models (LLMs) typically conflates two sources of variance: genuine model performance differences and measurement instability.<n>We investigate evaluation reliability by holding generation conditions constant while varying target language.<n>Our findings suggest that zero-shot judge transfer is unreliable for discourse-level assessment in morphologically rich languages.
  arXiv  Detail & Related papers  (2026-02-02T16:27:32Z)
• Narrowing the Gap between Zero- and Few-shot Machine Translation by Matching Styles [53.92189950211852]
  Large language models have demonstrated their ability to generalize to machine translation using zero- and few-shot examples with in-context learning. In this paper, we investigate the factors contributing to this gap and find that this gap can largely be closed (for about 70%) by matching the writing styles of the target corpus.
  arXiv  Detail & Related papers  (2023-11-04T03:18:45Z)
• Unlikelihood Tuning on Negative Samples Amazingly Improves Zero-Shot Translation [79.96416609433724]
  Zero-shot translation (ZST) aims to translate between unseen language pairs in training data. The common practice to guide the zero-shot language mapping during inference is to deliberately insert the source and target language IDs. Recent studies have shown that language IDs sometimes fail to navigate the ZST task, making them suffer from the off-target problem.
  arXiv  Detail & Related papers  (2023-09-28T17:02:36Z)
• On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluation [86.19634542434711]
  Mauve measures an information-theoretic divergence between two probability distributions over strings. We show that Mauve was right for the wrong reasons, and that its newly proposed divergence is not necessary for its high performance. We conclude that -- by encoding syntactic- and coherence-level features of text, while ignoring surface-level features -- such cluster-based substitutes to string distributions may simply be better for evaluating state-of-the-art language generators.
  arXiv  Detail & Related papers  (2022-05-31T17:58:49Z)
• OneAligner: Zero-shot Cross-lingual Transfer with One Rich-Resource Language Pair for Low-Resource Sentence Retrieval [91.76575626229824]
  We present OneAligner, an alignment model specially designed for sentence retrieval tasks. When trained with all language pairs of a large-scale parallel multilingual corpus (OPUS-100), this model achieves the state-of-the-art result. We conclude through empirical results and analyses that the performance of the sentence alignment task depends mostly on the monolingual and parallel data size.
  arXiv  Detail & Related papers  (2022-05-17T19:52:42Z)
• SimCSE: Simple Contrastive Learning of Sentence Embeddings [10.33373737281907]
  This paper presents SimCSE, a contrastive learning framework for embeddings. We first describe an unsupervised approach, which takes an input sentence and predicts itself in a contrastive objective. We then incorporate annotated pairs from NLI datasets into contrastive learning by using "entailment" pairs as positives and "contradiction" pairs as hard negatives.
  arXiv  Detail & Related papers  (2021-04-18T11:27:08Z)
• AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages [75.08199398141744]
  We present AmericasNLI, an extension of XNLI (Conneau et al.), to 10 indigenous languages of the Americas. We conduct experiments with XLM-R, testing multiple zero-shot and translation-based approaches. We find that XLM-R's zero-shot performance is poor for all 10 languages, with an average performance of 38.62%.
  arXiv  Detail & Related papers  (2021-04-18T05:32:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.