A Formal Limitation on Learning Human Language From Textual Corpora

• URL: http://arxiv.org/abs/2608.28560v1
• Date: Fri, 28 Aug 2026 17:38:24 GMT
• Title: A Formal Limitation on Learning Human Language From Textual Corpora
• Abstract summary: We derive upper bounds on the probability that a decoder recovers a speaker's intended meaning from a representation of the utterance.<n>Experiments on artificial languages, Mandarin zero-pronoun resolution, and color reference provide empirical evidence in support of the theory.
• Score: 54.22186267212778
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Can a listener recover what a speaker means from the form of an utterance alone? We answer this question information-theoretically, and for a listener given by any featurizer of text, including the hidden states of contemporary large language models. Modeling language use as a joint distribution over meanings, contexts, and utterances, we derive upper bounds on the probability that a decoder recovers a speaker's intended meaning from a representation of the utterance. The bounds are governed by the uncertainty that form leaves about meaning, which splits into an irreducible part and a part that only (extralinguistic) context, but never the utterance alone, can resolve. Because these quantities are intrinsic to language, no representation, however much text or supervision produced it, can surpass them; the bounds hold whether the space of meanings is discrete or continuous. Experiments on artificial languages, Mandarin zero-pronoun resolution, and color reference provide empirical evidence in support of the theory.

Related papers

• Generating in the Limit with Infinitely Many Hallucinations [86.48063079851867]
  Recently introduced framework of language generation in the limit requires the learner to produce valid, unseen strings from the target language.<n>We introduce a new notion of precision and recast this problem as the classic recall-precision trade-off.<n>We show that this relaxation can strictly increase recall when the adversary permanently withholds a large portion of the target language.
  arXiv  Detail & Related papers  (2026-06-08T09:58:13Z)
• Pixel Sentence Representation Learning [67.4775296225521]
  In this work, we conceptualize the learning of sentence-level textual semantics as a visual representation learning process. We employ visually-grounded text perturbation methods like typos and word order shuffling, resonating with human cognitive patterns, and enabling perturbation to be perceived as continuous. Our approach is further bolstered by large-scale unsupervised topical alignment training and natural language inference supervision.
  arXiv  Detail & Related papers  (2024-02-13T02:46:45Z)
• Multilingual Text Representation [3.4447129363520337]
  Modern NLP breakthrough includes large multilingual models capable of performing tasks across more than 100 languages. State-of-the-art language models came a long way, starting from the simple one-hot representation of words. We discuss how the full potential of language democratization could be obtained, reaching beyond the known limits.
  arXiv  Detail & Related papers  (2023-09-02T14:21:22Z)
• Natural Language Decompositions of Implicit Content Enable Better Text Representations [52.992875653864076]
  We introduce a method for the analysis of text that takes implicitly communicated content explicitly into account.<n>We use a large language model to produce sets of propositions that are inferentially related to the text that has been observed.<n>Our results suggest that modeling the meanings behind observed language, rather than the literal text alone, is a valuable direction for NLP.
  arXiv  Detail & Related papers  (2023-05-23T23:45:20Z)
• Learning an Artificial Language for Knowledge-Sharing in Multilingual Translation [15.32063273544696]
  We discretize the latent space of multilingual models by assigning encoder states to entries in a codebook. We validate our approach on large-scale experiments with realistic data volumes and domains. We also use the learned artificial language to analyze model behavior, and discover that using a similar bridge language increases knowledge-sharing among the remaining languages.
  arXiv  Detail & Related papers  (2022-11-02T17:14:42Z)
• Entailment Semantics Can Be Extracted from an Ideal Language Model [32.5500309433108]
  We prove that entailment judgments between sentences can be extracted from an ideal language model. We also show entailment judgments can be decoded from the predictions of a language model trained on such Gricean data.
  arXiv  Detail & Related papers  (2022-09-26T04:16:02Z)
• Testing the Ability of Language Models to Interpret Figurative Language [69.59943454934799]
  Figurative and metaphorical language are commonplace in discourse. It remains an open question to what extent modern language models can interpret nonliteral phrases. We introduce Fig-QA, a Winograd-style nonliteral language understanding task.
  arXiv  Detail & Related papers  (2022-04-26T23:42:22Z)
• Provable Limitations of Acquiring Meaning from Ungrounded Form: What will Future Language Models Understand? [87.20342701232869]
  We investigate the abilities of ungrounded systems to acquire meaning. We study whether assertions enable a system to emulate representations preserving semantic relations like equivalence. We find that assertions enable semantic emulation if all expressions in the language are referentially transparent. However, if the language uses non-transparent patterns like variable binding, we show that emulation can become an uncomputable problem.
  arXiv  Detail & Related papers  (2021-04-22T01:00:17Z)
• On the Optimality of Vagueness: "Around", "Between", and the Gricean Maxims [0.0]
  We argue that the use of vague expressions can offer an optimal tradeoff between truthfulness and informativeness. We focus on expressions of approximation such as "around", which are semantically vague. That is, vague sentences can be more informative than their precise counterparts.
  arXiv  Detail & Related papers  (2020-08-26T21:57:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.