Internal Knowledge Without External Expression: Probing the Generalization Boundary of a Classical Chinese Language Model

• URL: http://arxiv.org/abs/2604.14180v1
• Date: Tue, 31 Mar 2026 07:37:15 GMT
• Title: Internal Knowledge Without External Expression: Probing the Generalization Boundary of a Classical Chinese Language Model
• Abstract summary: We train a language model from scratch on a curated corpus of 1.56 billion tokens of pure Classical Chinese.<n>We investigate whether the model can distinguish known from unknown inputs.<n>We find a clear dissociation between internal and external uncertainty.
• Score: 2.096023520617773
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We train a 318M-parameter Transformer language model from scratch on a curated corpus of 1.56 billion tokens of pure Classical Chinese, with zero English characters or Arabic numerals. Through systematic out-of-distribution (OOD) testing, we investigate whether the model can distinguish known from unknown inputs, and crucially, whether it can express this distinction in its generated text. We find a clear dissociation between internal and external uncertainty. Internally, the model exhibits a perplexity jump ratio of 2.39x between real and fabricated historical events (p = 8.9e-11, n = 92 per group), with semi-fabricated events (real figures + fictional events) showing the highest perplexity (4.24x, p = 1.1e-16), demonstrating genuine factual encoding beyond syntactic pattern matching. Externally, however, the model never learns to express uncertainty: classical Chinese epistemic markers appear at lower rates for OOD questions (3.5%) than for in-distribution questions (8.3%, p = 0.023), reflecting rhetorical conventions rather than genuine metacognition. We replicate both findings across three languages (Classical Chinese, English, Japanese), three writing systems, and eight models from 110M to 1.56B parameters. We further show that uncertainty expression frequency is determined entirely by training data conventions, with Classical Chinese models showing a "humility paradox" (more hedging for known topics), while Japanese models almost never hedge. We argue that metacognitive expression -- the ability to say "I don't know" -- does not emerge from language modeling alone and requires explicit training signals such as RLHF.

Related papers

• How China-Origin Vision-Language Models Move from Refusal to Reframing in State Alignment [8.359235542667806]
  State-aligned distortion has been documented in China-origin text-based large language models (LLMs)<n>We run nine vision-language models (VLMs), seven China-origin and two non-China, across four elicitation paradigms and two prompt languages, yielding 21,708 trials.<n>Each response is audited on six dimensions -- explicit refusal, information integrity, visual grounding, state-aligned framing, language consistency, and response length.
  arXiv  Detail & Related papers  (2026-08-12T08:58:37Z)
• We Hebben Een Serieus Translatie: Modeling Intercomprehension as Probabilistic Inference [10.293916944634804]
  Intercomprehension refers to partial intelligibility of an unfamiliar language by a speaker of a related language.<n>In this work, we extend past work on algorithmic models of noisy-channel inference to model intercomprehension in a Bayesian framework.<n>We conduct a human behavioral experiment, eliciting inferences for utterances in Dutch, Italian, and Ukrainian from speakers of English, Spanish, and Russian, respectively.
  arXiv  Detail & Related papers  (2026-07-13T21:30:05Z)
• The Rise of Verbal Tics in Large Language Models: A Systematic Analysis Across Frontier Models [12.320824168302908]
  We present a systematic analysis of the verbal tic phenomenon across eight state-of-the-art Large Language Models (LLMs)<n>We introduce the Verbal Tic Index (VTI), a composite metric quantifying tic prevalence, and analyze its correlation with sycophancy, lexical diversity, and human-perceived naturalness.
  arXiv  Detail & Related papers  (2026-04-21T06:43:01Z)
• The Randomness Floor: Measuring Intrinsic Non-Randomness in Language Model Token Distributions [0.0]
  This paper introduces Entropic Deviation (ED), the normalised KL divergence between a model's token distribution and the uniform distribution.<n>Three transformer families converge on nearly identical ED values despite different training data and vocabularies.<n>Cross-lingual experiments with Qwen-32B show a stable gradient across five languages that does not correlate with token fertility.
  arXiv  Detail & Related papers  (2026-03-29T21:17:46Z)
• Repetition Without Exclusivity: Scale Sensitivity of Referential Mechanisms in Child-Scale Language Models [0.0]
  We operationalise mutual exclusivity as referential suppression in language models trained on child-directed speech.<n>We find that distributional learning on child-directed speech produces repetition-based reference tracking rather than lexical exclusivity.<n>We argue that referential grounding may be a necessary ingredient for ME -- an empirical claim about required input structure, not a nativist one.
  arXiv  Detail & Related papers  (2026-03-14T01:59:50Z)
• Unmasking the Factual-Conceptual Gap in Persian Language Models [0.8180770164225369]
  We introduce DivanBench, a benchmark focused on superstitions and customs, arbitrary, context-dependent rules that resist simple logical deduction.<n>We evaluate seven Persian LLMs and reveal three critical failures: most models exhibit severe acquiescence bias, correctly identifying appropriate behaviors but failing to reject clear violations.
  arXiv  Detail & Related papers  (2026-02-19T18:42:46Z)
• DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generation [111.94720088481614]
  Can multimodal generative models effectively produce content given dialectal textual input?<n>We construct a new large-scale benchmark spanning six common English dialects.<n>We design a general encoder-based mitigation strategy for multimodal generative models.
  arXiv  Detail & Related papers  (2025-10-16T17:56:55Z)
• CLEAR-3K: Assessing Causal Explanatory Capabilities in Language Models [3.137688620241855]
  We introduce CLEAR-3K, a dataset of 3,000 assertion-reasoning questions designed to evaluate whether language models can determine if one statement causally explains another.<n>Each question present an assertion-reason pair and challenge language models to distinguish between semantic relatedness and genuine causal explanatory relationships.
  arXiv  Detail & Related papers  (2025-06-20T17:35:36Z)
• ZhoBLiMP: a Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinese [18.90198316015052]
  We first introduce ZhoBLiMP, the most comprehensive benchmark of linguistic minimal pairs for Chinese. We then train 20 LMs of different sizes (14M to 1.4B) on Chinese corpora and evaluate them along with 14 off-the-shelf LLMs on ZhoBLiMP. The overall results indicate that Chinese grammar can be mostly learned by models with around 500M parameters, trained on 1B tokens with one epoch, showing limited benefits for further scaling.
  arXiv  Detail & Related papers  (2024-11-09T07:16:08Z)
• Perceptions of Linguistic Uncertainty by Language Models and Humans [26.69714008538173]
  We investigate how language models map linguistic expressions of uncertainty to numerical responses. We find that 7 out of 10 models are able to map uncertainty expressions to probabilistic responses in a human-like manner. This sensitivity indicates that language models are substantially more susceptible to bias based on their prior knowledge.
  arXiv  Detail & Related papers  (2024-07-22T17:26:12Z)
• PanGu-{\Sigma}: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing [64.53242758625922]
  PanGu-Sigma is trained on a cluster of Ascend 910 AI processors and MindSpore framework. It provides state-of-the-art performance in zero-shot learning of various Chinese NLP downstream tasks.
  arXiv  Detail & Related papers  (2023-03-20T03:39:27Z)
• Scaling Laws for Generative Mixed-Modal Language Models [103.25737824352949]
  We report new mixed-modal scaling laws that unify the contributions of individual modalities and the interactions between them. Specifically, we explicitly model the optimal synergy and competition due to data and model size as an additive term to previous uni-modal scaling laws. We also find four empirical phenomena observed during the training, such as emergent coordinate-ascent style training that naturally alternates between modalities.
  arXiv  Detail & Related papers  (2023-01-10T00:20:06Z)
• Discovering Latent Knowledge in Language Models Without Supervision [72.95136739040676]
  Existing techniques for training language models can be misaligned with the truth. We propose directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models.
  arXiv  Detail & Related papers  (2022-12-07T18:17:56Z)
• Few-shot Learning with Multilingual Language Models [66.49496434282564]
  We train multilingual autoregressive language models on a balanced corpus covering a diverse set of languages. Our largest model sets new state of the art in few-shot learning in more than 20 representative languages. We present a detailed analysis of where the model succeeds and fails, showing in particular that it enables cross-lingual in-context learning.
  arXiv  Detail & Related papers  (2021-12-20T16:52:35Z)
• Multi-timescale Representation Learning in LSTM Language Models [69.98840820213937]
  Language models must capture statistical dependencies between words at timescales ranging from very short to very long. We derived a theory for how the memory gating mechanism in long short-term memory language models can capture power law decay. Experiments showed that LSTM language models trained on natural English text learn to approximate this theoretical distribution.
  arXiv  Detail & Related papers  (2020-09-27T02:13:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.