A Study of Compositional Generalization in Neural Models
- URL: http://arxiv.org/abs/2006.09437v2
- Date: Wed, 8 Jul 2020 15:50:41 GMT
- Title: A Study of Compositional Generalization in Neural Models
- Authors: Tim Klinger, Dhaval Adjodah, Vincent Marois, Josh Joseph, Matthew
Riemer, Alex 'Sandy' Pentland, Murray Campbell
- Abstract summary: We introduce ConceptWorld, which enables the generation of images from compositional and relational concepts.
We perform experiments to test the ability of standard neural networks to generalize on relations with compositional arguments.
For simple problems, all models generalize well to close concepts but struggle with longer compositional chains.
- Score: 22.66002315559978
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Compositional and relational learning is a hallmark of human intelligence,
but one which presents challenges for neural models. One difficulty in the
development of such models is the lack of benchmarks with clear compositional
and relational task structure on which to systematically evaluate them. In this
paper, we introduce an environment called ConceptWorld, which enables the
generation of images from compositional and relational concepts, defined using
a logical domain specific language. We use it to generate images for a variety
of compositional structures: 2x2 squares, pentominoes, sequences, scenes
involving these objects, and other more complex concepts. We perform
experiments to test the ability of standard neural architectures to generalize
on relations with compositional arguments as the compositional depth of those
arguments increases and under substitution. We compare standard neural networks
such as MLP, CNN and ResNet, as well as state-of-the-art relational networks
including WReN and PrediNet in a multi-class image classification setting. For
simple problems, all models generalize well to close concepts but struggle with
longer compositional chains. For more complex tests involving substitutivity,
all models struggle, even with short chains. In highlighting these difficulties
and providing an environment for further experimentation, we hope to encourage
the development of models which are able to generalize effectively in
compositional, relational domains.
Related papers
- What makes Models Compositional? A Theoretical View: With Supplement [60.284698521569936]
We propose a general neuro-symbolic definition of compositional functions and their compositional complexity.
We show how various existing general and special purpose sequence processing models fit this definition and use it to analyze their compositional complexity.
arXiv Detail & Related papers (2024-05-02T20:10:27Z) - Foundational Models Defining a New Era in Vision: A Survey and Outlook [151.49434496615427]
Vision systems to see and reason about the compositional nature of visual scenes are fundamental to understanding our world.
The models learned to bridge the gap between such modalities coupled with large-scale training data facilitate contextual reasoning, generalization, and prompt capabilities at test time.
The output of such models can be modified through human-provided prompts without retraining, e.g., segmenting a particular object by providing a bounding box, having interactive dialogues by asking questions about an image or video scene or manipulating the robot's behavior through language instructions.
arXiv Detail & Related papers (2023-07-25T17:59:18Z) - Compositional diversity in visual concept learning [18.907108368038216]
Humans leverage compositionality to efficiently learn new concepts, understanding how familiar parts can combine together to form novel objects.
Here, we study how people classify and generate alien figures'' with rich relational structure.
We develop a Bayesian program induction model which searches for the best programs for generating the candidate visual figures.
arXiv Detail & Related papers (2023-05-30T19:30:50Z) - Compositional Processing Emerges in Neural Networks Solving Math
Problems [100.80518350845668]
Recent progress in artificial neural networks has shown that when large models are trained on enough linguistic data, grammatical structure emerges in their representations.
We extend this work to the domain of mathematical reasoning, where it is possible to formulate precise hypotheses about how meanings should be composed.
Our work shows that neural networks are not only able to infer something about the structured relationships implicit in their training data, but can also deploy this knowledge to guide the composition of individual meanings into composite wholes.
arXiv Detail & Related papers (2021-05-19T07:24:42Z) - Learning Graph Embeddings for Compositional Zero-shot Learning [73.80007492964951]
In compositional zero-shot learning, the goal is to recognize unseen compositions of observed visual primitives states.
We propose a novel graph formulation called Compositional Graph Embedding (CGE) that learns image features and latent representations of visual primitives in an end-to-end manner.
By learning a joint compatibility that encodes semantics between concepts, our model allows for generalization to unseen compositions without relying on an external knowledge base like WordNet.
arXiv Detail & Related papers (2021-02-03T10:11:03Z) - Counterfactual Generative Networks [59.080843365828756]
We propose to decompose the image generation process into independent causal mechanisms that we train without direct supervision.
By exploiting appropriate inductive biases, these mechanisms disentangle object shape, object texture, and background.
We show that the counterfactual images can improve out-of-distribution with a marginal drop in performance on the original classification task.
arXiv Detail & Related papers (2021-01-15T10:23:12Z) - Compositionally Generalizable 3D Structure Prediction [41.641683644620464]
Single-image 3D shape reconstruction is an important and long-standing problem in computer vision.
We propose a novel framework that could better generalize to unseen object categories.
Experiments on PartNet show that we achieve superior performance than state-of-the-art.
arXiv Detail & Related papers (2020-12-04T09:53:14Z) - Learning Task-General Representations with Generative Neuro-Symbolic
Modeling [22.336243882030026]
We develop a generative neuro-symbolic (GNS) model of handwritten character concepts.
The correlations between parts are modeled with neural network subroutines, allowing the model to learn directly from raw data.
In a subsequent evaluation, our GNS model uses probabilistic inference to learn rich conceptual representations from a single training image.
arXiv Detail & Related papers (2020-06-25T14:41:27Z) - Compositional Generalization by Learning Analytical Expressions [87.15737632096378]
A memory-augmented neural model is connected with analytical expressions to achieve compositional generalization.
Experiments on the well-known benchmark SCAN demonstrate that our model seizes a great ability of compositional generalization.
arXiv Detail & Related papers (2020-06-18T15:50:57Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.