Language-Based Causal Representation Learning
- URL: http://arxiv.org/abs/2207.05259v1
- Date: Tue, 12 Jul 2022 02:07:58 GMT
- Title: Language-Based Causal Representation Learning
- Authors: Blai Bonet and Hector Geffner
- Abstract summary: We show that the dynamics is learned over a suitable domain-independent first-order causal language.
The preference for the most compact representation in the language that is compatible with the data provides a strong and meaningful learning bias.
While "classical AI" requires handcrafted representations, similar representations can be learned from unstructured data over the same languages.
- Score: 24.008923963650226
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Consider the finite state graph that results from a simple, discrete,
dynamical system in which an agent moves in a rectangular grid picking up and
dropping packages. Can the state variables of the problem, namely, the agent
location and the package locations, be recovered from the structure of the
state graph alone without having access to information about the objects, the
structure of the states, or any background knowledge? We show that this is
possible provided that the dynamics is learned over a suitable
domain-independent first-order causal language that makes room for objects and
relations that are not assumed to be known. The preference for the most compact
representation in the language that is compatible with the data provides a
strong and meaningful learning bias that makes this possible. The language of
structured causal models (SCMs) is the standard language for representing
(static) causal models but in dynamic worlds populated by objects, first-order
causal languages such as those used in "classical AI planning" are required.
While "classical AI" requires handcrafted representations, similar
representations can be learned from unstructured data over the same languages.
Indeed, it is the languages and the preference for compact representations in
those languages that provide structure to the world, uncovering objects,
relations, and causes.
Related papers
- Improving Arithmetic Reasoning Ability of Large Language Models through Relation Tuples, Verification and Dynamic Feedback [14.938401898546553]
We propose to use a semi-structured form to represent reasoning steps of large language models.
Specifically, we use relations, which are not only human but also machine-friendly and easier to verify than natural language.
arXiv Detail & Related papers (2024-06-25T18:21:00Z) - Enhancing Language Representation with Constructional Information for
Natural Language Understanding [5.945710973349298]
We introduce construction grammar (CxG), which highlights the pairings of form and meaning.
We adopt usage-based construction grammar as the basis of our work.
A HyCxG framework is proposed to enhance language representation through a three-stage solution.
arXiv Detail & Related papers (2023-06-05T12:15:12Z) - Physics of Language Models: Part 1, Learning Hierarchical Language Structures [51.68385617116854]
Transformer-based language models are effective but complex, and understanding their inner workings is a significant challenge.
We introduce a family of synthetic CFGs that produce hierarchical rules, capable of generating lengthy sentences.
We demonstrate that generative models like GPT can accurately learn this CFG language and generate sentences based on it.
arXiv Detail & Related papers (2023-05-23T04:28:16Z) - Knowledge Graph Guided Semantic Evaluation of Language Models For User
Trust [7.063958622970576]
This study evaluates the encoded semantics in the self-attention transformers by leveraging explicit knowledge graph structures.
The opacity of language models has an immense bearing on societal issues of trust and explainable decision outcomes.
arXiv Detail & Related papers (2023-05-08T18:53:14Z) - Benchmarking Language Models for Code Syntax Understanding [79.11525961219591]
Pre-trained language models have demonstrated impressive performance in both natural language processing and program understanding.
In this work, we perform the first thorough benchmarking of the state-of-the-art pre-trained models for identifying the syntactic structures of programs.
Our findings point out key limitations of existing pre-training methods for programming languages, and suggest the importance of modeling code syntactic structures.
arXiv Detail & Related papers (2022-10-26T04:47:18Z) - Linking Emergent and Natural Languages via Corpus Transfer [98.98724497178247]
We propose a novel way to establish a link by corpus transfer between emergent languages and natural languages.
Our approach showcases non-trivial transfer benefits for two different tasks -- language modeling and image captioning.
We also introduce a novel metric to predict the transferability of an emergent language by translating emergent messages to natural language captions grounded on the same images.
arXiv Detail & Related papers (2022-03-24T21:24:54Z) - Pre-Trained Language Models for Interactive Decision-Making [72.77825666035203]
We describe a framework for imitation learning in which goals and observations are represented as a sequence of embeddings.
We demonstrate that this framework enables effective generalization across different environments.
For test tasks involving novel goals or novel scenes, initializing policies with language models improves task completion rates by 43.6%.
arXiv Detail & Related papers (2022-02-03T18:55:52Z) - Towards Zero-shot Language Modeling [90.80124496312274]
We construct a neural model that is inductively biased towards learning human languages.
We infer this distribution from a sample of typologically diverse training languages.
We harness additional language-specific side information as distant supervision for held-out languages.
arXiv Detail & Related papers (2021-08-06T23:49:18Z) - Linguistic Inspired Graph Analysis [0.0]
Isomorphisms allow human cognition to transcribe a potentially unsolvable problem from one domain to a different domain.
Current approaches only focus on transcribing structural information from the source to target structure.
It is found that further work needs to be done to understand how graphs can be enriched to allow for isomorphisms to capture semantic and pragmatic information.
arXiv Detail & Related papers (2021-05-13T12:16:30Z) - Graph-Structured Referring Expression Reasoning in The Wild [105.95488002374158]
Grounding referring expressions aims to locate in an image an object referred to by a natural language expression.
We propose a scene graph guided modular network (SGMN) to perform reasoning over a semantic graph and a scene graph.
We also propose Ref-Reasoning, a large-scale real-world dataset for structured referring expression reasoning.
arXiv Detail & Related papers (2020-04-19T11:00:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.