Generalized Optimal Linear Orders
- URL: http://arxiv.org/abs/2108.10692v1
- Date: Fri, 13 Aug 2021 13:10:15 GMT
- Title: Generalized Optimal Linear Orders
- Authors: Rishi Bommasani
- Abstract summary: The sequential structure of language, and the order of words in a sentence specifically, plays a central role in human language processing.
In designing computational models of language, the de facto approach is to present sentences to machines with the words ordered in the same order as in the original human-authored sentence.
The very essence of this work is to question the implicit assumption that this is desirable and inject theoretical soundness into the consideration of word order in natural language processing.
- Score: 9.010643838773477
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The sequential structure of language, and the order of words in a sentence
specifically, plays a central role in human language processing. Consequently,
in designing computational models of language, the de facto approach is to
present sentences to machines with the words ordered in the same order as in
the original human-authored sentence. The very essence of this work is to
question the implicit assumption that this is desirable and inject theoretical
soundness into the consideration of word order in natural language processing.
In this thesis, we begin by uniting the disparate treatments of word order in
cognitive science, psycholinguistics, computational linguistics, and natural
language processing under a flexible algorithmic framework. We proceed to use
this heterogeneous theoretical foundation as the basis for exploring new word
orders with an undercurrent of psycholinguistic optimality. In particular, we
focus on notions of dependency length minimization given the difficulties in
human and computational language processing in handling long-distance
dependencies. We then discuss algorithms for finding optimal word orders
efficiently in spite of the combinatorial space of possibilities. We conclude
by addressing the implications of these word orders on human language and their
downstream impacts when integrated in computational models.
Related papers
- Linguistic Structure from a Bottleneck on Sequential Information Processing [5.850665541267672]
We show that natural-language-like systematicity arises in codes that are constrained by predictive information.
We show that human languages are structured to have low predictive information at the levels of phonology, morphology, syntax, and semantics.
arXiv Detail & Related papers (2024-05-20T15:25:18Z) - Nondeterministic Stacks in Neural Networks [0.456877715768796]
We develop a differentiable data structure that efficiently simulates a nondeterministic pushdown automaton.
We show that this raises their formal recognition power to arbitrary context-free languages.
We also show that an RNN augmented with a nondeterministic stack is capable of surprisingly powerful behavior.
arXiv Detail & Related papers (2023-04-25T16:00:40Z) - Word class representations spontaneously emerge in a deep neural network
trained on next word prediction [7.240611820374677]
How do humans learn language, and can the first language be learned at all?
These fundamental questions are still hotly debated.
In particular, we train an artificial deep neural network on predicting the next word.
We find that the internal representations of nine-word input sequences cluster according to the word class of the tenth word to be predicted as output.
arXiv Detail & Related papers (2023-02-15T11:02:50Z) - Categorical Tools for Natural Language Processing [0.0]
This thesis develops the translation between category theory and computational linguistics.
The three chapters deal with syntax, semantics and pragmatics.
The resulting functorial models can be composed to form games where equilibria are the solutions of language processing tasks.
arXiv Detail & Related papers (2022-12-13T15:12:37Z) - Benchmarking Language Models for Code Syntax Understanding [79.11525961219591]
Pre-trained language models have demonstrated impressive performance in both natural language processing and program understanding.
In this work, we perform the first thorough benchmarking of the state-of-the-art pre-trained models for identifying the syntactic structures of programs.
Our findings point out key limitations of existing pre-training methods for programming languages, and suggest the importance of modeling code syntactic structures.
arXiv Detail & Related papers (2022-10-26T04:47:18Z) - Context based lemmatizer for Polish language [0.0]
Lemmatization is the process of grouping together the inflected forms of a word so they can be analysed as a single item.
In computational linguistics, lemmatisation is the algorithmic process of determining the lemma of a word based on its intended meaning.
The model achieves the best results for polish language lemmatisation process.
arXiv Detail & Related papers (2022-07-23T18:02:16Z) - Neural Abstructions: Abstractions that Support Construction for Grounded
Language Learning [69.1137074774244]
Leveraging language interactions effectively requires addressing limitations in the two most common approaches to language grounding.
We introduce the idea of neural abstructions: a set of constraints on the inference procedure of a label-conditioned generative model.
We show that with this method a user population is able to build a semantic modification for an open-ended house task in Minecraft.
arXiv Detail & Related papers (2021-07-20T07:01:15Z) - Low-Dimensional Structure in the Space of Language Representations is
Reflected in Brain Responses [62.197912623223964]
We show a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings.
We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI.
This suggests that the embedding captures some part of the brain's natural language representation structure.
arXiv Detail & Related papers (2021-06-09T22:59:12Z) - Lexically-constrained Text Generation through Commonsense Knowledge
Extraction and Injection [62.071938098215085]
We focus on the Commongen benchmark, wherein the aim is to generate a plausible sentence for a given set of input concepts.
We propose strategies for enhancing the semantic correctness of the generated text.
arXiv Detail & Related papers (2020-12-19T23:23:40Z) - Infusing Finetuning with Semantic Dependencies [62.37697048781823]
We show that, unlike syntax, semantics is not brought to the surface by today's pretrained models.
We then use convolutional graph encoders to explicitly incorporate semantic parses into task-specific finetuning.
arXiv Detail & Related papers (2020-12-10T01:27:24Z) - Semantics-Aware Inferential Network for Natural Language Understanding [79.70497178043368]
We propose a Semantics-Aware Inferential Network (SAIN) to meet such a motivation.
Taking explicit contextualized semantics as a complementary input, the inferential module of SAIN enables a series of reasoning steps over semantic clues.
Our model achieves significant improvement on 11 tasks including machine reading comprehension and natural language inference.
arXiv Detail & Related papers (2020-04-28T07:24:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.