Modelling Compositionality and Structure Dependence in Natural Language
- URL: http://arxiv.org/abs/2012.02038v2
- Date: Wed, 30 Dec 2020 17:14:13 GMT
- Title: Modelling Compositionality and Structure Dependence in Natural Language
- Authors: Karthikeya Ramesh Kaushik, Andrea E. Martin
- Abstract summary: Drawing on linguistics and set theory, a formalisation of these ideas is presented in the first half of this thesis.
We see how cognitive systems that process language need to have certain functional constraints.
Using the advances of word embedding techniques, a model of relational learning is simulated.
- Score: 0.12183405753834563
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Human beings possess the most sophisticated computational machinery in the
known universe. We can understand language of rich descriptive power, and
communicate in the same environment with astonishing clarity. Two of the many
contributors to the interest in natural language - the properties of
Compositionality and Structure Dependence, are well documented, and offer a
vast space to ask interesting modelling questions. The first step to begin
answering these questions is to ground verbal theory in formal terms. Drawing
on linguistics and set theory, a formalisation of these ideas is presented in
the first half of this thesis. We see how cognitive systems that process
language need to have certain functional constraints, viz. time based,
incremental operations that rely on a structurally defined domain. The
observations that result from analysing this formal setup are examined as part
of a modelling exercise. Using the advances of word embedding techniques, a
model of relational learning is simulated with a custom dataset to demonstrate
how a time based role-filler binding mechanism satisfies some of the
constraints described in the first section. The model's ability to map
structure, along with its symbolic-connectionist architecture makes for a
cognitively plausible implementation. The formalisation and simulation are
together an attempt to recognise the constraints imposed by linguistic theory,
and explore the opportunities presented by a cognitive model of relation
learning to realise these constraints.
Related papers
- A Complexity-Based Theory of Compositionality [53.025566128892066]
In AI, compositional representations can enable a powerful form of out-of-distribution generalization.
Here, we propose a formal definition of compositionality that accounts for and extends our intuitions about compositionality.
The definition is conceptually simple, quantitative, grounded in algorithmic information theory, and applicable to any representation.
arXiv Detail & Related papers (2024-10-18T18:37:27Z) - A Survey on Compositional Learning of AI Models: Theoretical and Experimetnal Practices [15.92779896185647]
Compositional learning is crucial for human cognition, especially in human language comprehension and visual perception.
Despite its integral role in intelligence, there is a lack of systematic theoretical and experimental research methodologies.
This paper surveys the literature on compositional learning of AI models and the connections made to cognitive studies.
arXiv Detail & Related papers (2024-06-13T03:46:21Z) - Language Evolution with Deep Learning [49.879239655532324]
Computational modeling plays an essential role in the study of language emergence.
It aims to simulate the conditions and learning processes that could trigger the emergence of a structured language.
This chapter explores another class of computational models that have recently revolutionized the field of machine learning: deep learning models.
arXiv Detail & Related papers (2024-03-18T16:52:54Z) - Learning Interpretable Concepts: Unifying Causal Representation Learning
and Foundation Models [51.43538150982291]
We study how to learn human-interpretable concepts from data.
Weaving together ideas from both fields, we show that concepts can be provably recovered from diverse data.
arXiv Detail & Related papers (2024-02-14T15:23:59Z) - Foundational Models Defining a New Era in Vision: A Survey and Outlook [151.49434496615427]
Vision systems to see and reason about the compositional nature of visual scenes are fundamental to understanding our world.
The models learned to bridge the gap between such modalities coupled with large-scale training data facilitate contextual reasoning, generalization, and prompt capabilities at test time.
The output of such models can be modified through human-provided prompts without retraining, e.g., segmenting a particular object by providing a bounding box, having interactive dialogues by asking questions about an image or video scene or manipulating the robot's behavior through language instructions.
arXiv Detail & Related papers (2023-07-25T17:59:18Z) - From Word Models to World Models: Translating from Natural Language to
the Probabilistic Language of Thought [124.40905824051079]
We propose rational meaning construction, a computational framework for language-informed thinking.
We frame linguistic meaning as a context-sensitive mapping from natural language into a probabilistic language of thought.
We show that LLMs can generate context-sensitive translations that capture pragmatically-appropriate linguistic meanings.
We extend our framework to integrate cognitively-motivated symbolic modules.
arXiv Detail & Related papers (2023-06-22T05:14:00Z) - Feature Interactions Reveal Linguistic Structure in Language Models [2.0178765779788495]
We study feature interactions in the context of feature attribution methods for post-hoc interpretability.
We work out a grey box methodology, in which we train models to perfection on a formal language classification task.
We show that under specific configurations, some methods are indeed able to uncover the grammatical rules acquired by a model.
arXiv Detail & Related papers (2023-06-21T11:24:41Z) - Multi-Relational Hyperbolic Word Embeddings from Natural Language
Definitions [5.763375492057694]
This paper presents a multi-relational model that explicitly leverages such a structure to derive word embeddings from definitions.
An empirical analysis demonstrates that the framework can help imposing the desired structural constraints.
Experiments reveal the superiority of the Hyperbolic word embeddings over the Euclidean counterparts.
arXiv Detail & Related papers (2023-05-12T08:16:06Z) - Compositional Processing Emerges in Neural Networks Solving Math
Problems [100.80518350845668]
Recent progress in artificial neural networks has shown that when large models are trained on enough linguistic data, grammatical structure emerges in their representations.
We extend this work to the domain of mathematical reasoning, where it is possible to formulate precise hypotheses about how meanings should be composed.
Our work shows that neural networks are not only able to infer something about the structured relationships implicit in their training data, but can also deploy this knowledge to guide the composition of individual meanings into composite wholes.
arXiv Detail & Related papers (2021-05-19T07:24:42Z) - Towards Coinductive Models for Natural Language Understanding. Bringing
together Deep Learning and Deep Semantics [0.0]
Coinduction has been successfully used in the design of operating systems and programming languages.
It has been present in text mining, machine translation, and in some attempts to model intensionality and modalities.
This article shows several examples of the joint appearance of induction and coinduction in natural language processing.
arXiv Detail & Related papers (2020-12-09T03:10:36Z) - Knowledge Patterns [19.57676317580847]
This paper describes a new technique, called "knowledge patterns", for helping construct axiom-rich, formal Ontology.
Knowledge patterns provide an important insight into the structure of a formal Ontology.
We describe the technique and an application built using them, and then critique their strengths and weaknesses.
arXiv Detail & Related papers (2020-05-08T22:33:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.