The Singleton Fallacy: Why Current Critiques of Language Models Miss the
Point
- URL: http://arxiv.org/abs/2102.04310v1
- Date: Mon, 8 Feb 2021 16:12:36 GMT
- Title: The Singleton Fallacy: Why Current Critiques of Language Models Miss the
Point
- Authors: Magnus Sahlgren, Fredrik Carlsson
- Abstract summary: We discuss the current critique against neural network-based Natural Language Understanding (NLU) solutions known as language models.
We will argue that there are many different types of language use, meaning and understanding, and that (current) language models are build with the explicit purpose of acquiring and representing one type of structural understanding of language.
- Score: 3.096615629099618
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper discusses the current critique against neural network-based
Natural Language Understanding (NLU) solutions known as language models. We
argue that much of the current debate rests on an argumentation error that we
will refer to as the singleton fallacy: the assumption that language, meaning,
and understanding are single and uniform phenomena that are unobtainable by
(current) language models. By contrast, we will argue that there are many
different types of language use, meaning and understanding, and that (current)
language models are build with the explicit purpose of acquiring and
representing one type of structural understanding of language. We will argue
that such structural understanding may cover several different modalities, and
as such can handle several different types of meaning. Our position is that we
currently see no theoretical reason why such structural knowledge would be
insufficient to count as "real" understanding.
Related papers
- Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons [43.708431369382176]
We introduce a small challenge dataset for NLI with large lexical overlap.
We show that GPT-4 and Llama 2 fail it with strong bias.
From a Computational Linguistics perspective, we identify a group of constructions with three classes of adjectives which cannot be distinguished by surface features.
arXiv Detail & Related papers (2024-03-26T14:51:12Z) - On General Language Understanding [18.2932386988379]
This paper sketches the outlines of a model of understanding, which can ground questions of the adequacy of current methods of measurement of model quality.
The paper makes three claims: A) That different language use situation types have different characteristics, B) That language understanding is a multifaceted phenomenon, and C) That the choice of Understanding Indicator marks the limits of benchmarking.
arXiv Detail & Related papers (2023-10-27T10:36:54Z) - From Word Models to World Models: Translating from Natural Language to
the Probabilistic Language of Thought [124.40905824051079]
We propose rational meaning construction, a computational framework for language-informed thinking.
We frame linguistic meaning as a context-sensitive mapping from natural language into a probabilistic language of thought.
We show that LLMs can generate context-sensitive translations that capture pragmatically-appropriate linguistic meanings.
We extend our framework to integrate cognitively-motivated symbolic modules.
arXiv Detail & Related papers (2023-06-22T05:14:00Z) - What does the Failure to Reason with "Respectively" in Zero/Few-Shot
Settings Tell Us about Language Models? [5.431715810374623]
We show how language models (LMs) reason with respective readings from two perspectives: syntactic-semantic and commonsense-world knowledge.
We show that fine-tuned NLI models struggle with understanding such readings without explicit supervision.
arXiv Detail & Related papers (2023-05-31T06:45:09Z) - Transparency Helps Reveal When Language Models Learn Meaning [71.96920839263457]
Our systematic experiments with synthetic data reveal that, with languages where all expressions have context-independent denotations, both autoregressive and masked language models learn to emulate semantic relations between expressions.
Turning to natural language, our experiments with a specific phenomenon -- referential opacity -- add to the growing body of evidence that current language models do not well-represent natural language semantics.
arXiv Detail & Related papers (2022-10-14T02:35:19Z) - Machine Reading, Fast and Slow: When Do Models "Understand" Language? [59.897515617661874]
We investigate the behavior of reading comprehension models with respect to two linguistic'skills': coreference resolution and comparison.
We find that for comparison (but not coreference) the systems based on larger encoders are more likely to rely on the 'right' information.
arXiv Detail & Related papers (2022-09-15T16:25:44Z) - Norm Participation Grounds Language [16.726800816202033]
I propose a different, and more wide-ranging conception of how grounding should be understood: What grounds language is its normative nature.
There are standards for doing things right, these standards are public and authoritative, while at the same time acceptance of authority can be disputed and negotiated.
What grounds language, then, is the determined use that language users make of it, and what it is grounded in is the community of language users.
arXiv Detail & Related papers (2022-06-06T20:21:59Z) - Testing the Ability of Language Models to Interpret Figurative Language [69.59943454934799]
Figurative and metaphorical language are commonplace in discourse.
It remains an open question to what extent modern language models can interpret nonliteral phrases.
We introduce Fig-QA, a Winograd-style nonliteral language understanding task.
arXiv Detail & Related papers (2022-04-26T23:42:22Z) - Interpreting Language Models with Contrastive Explanations [99.7035899290924]
Language models must consider various features to predict a token, such as its part of speech, number, tense, or semantics.
Existing explanation methods conflate evidence for all these features into a single explanation, which is less interpretable for human understanding.
We show that contrastive explanations are quantifiably better than non-contrastive explanations in verifying major grammatical phenomena.
arXiv Detail & Related papers (2022-02-21T18:32:24Z) - Provable Limitations of Acquiring Meaning from Ungrounded Form: What
will Future Language Models Understand? [87.20342701232869]
We investigate the abilities of ungrounded systems to acquire meaning.
We study whether assertions enable a system to emulate representations preserving semantic relations like equivalence.
We find that assertions enable semantic emulation if all expressions in the language are referentially transparent.
However, if the language uses non-transparent patterns like variable binding, we show that emulation can become an uncomputable problem.
arXiv Detail & Related papers (2021-04-22T01:00:17Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.