Learning Representations that Support Extrapolation
- URL: http://arxiv.org/abs/2007.05059v3
- Date: Wed, 6 Sep 2023 18:20:08 GMT
- Title: Learning Representations that Support Extrapolation
- Authors: Taylor W. Webb, Zachary Dulberg, Steven M. Frankland, Alexander A.
Petrov, Randall C. O'Reilly, Jonathan D. Cohen
- Abstract summary: We consider the challenge of learning representations that support extrapolation.
We introduce a novel visual analogy benchmark that allows the graded evaluation of extrapolation.
We also introduce a simple technique, temporal context normalization, that encourages representations that emphasize the relations between objects.
- Score: 39.84463809100903
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Extrapolation -- the ability to make inferences that go beyond the scope of
one's experiences -- is a hallmark of human intelligence. By contrast, the
generalization exhibited by contemporary neural network algorithms is largely
limited to interpolation between data points in their training corpora. In this
paper, we consider the challenge of learning representations that support
extrapolation. We introduce a novel visual analogy benchmark that allows the
graded evaluation of extrapolation as a function of distance from the convex
domain defined by the training data. We also introduce a simple technique,
temporal context normalization, that encourages representations that emphasize
the relations between objects. We find that this technique enables a
significant improvement in the ability to extrapolate, considerably
outperforming a number of competitive techniques.
Related papers
- Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond [61.18736646013446]
In pursuit of a deeper understanding of its surprising behaviors, we investigate the utility of a simple yet accurate model of a trained neural network.
Across three case studies, we illustrate how it can be applied to derive new empirical insights on a diverse range of prominent phenomena.
arXiv Detail & Related papers (2024-10-31T22:54:34Z) - The Relational Bottleneck as an Inductive Bias for Efficient Abstraction [3.19883356005403]
We show that neural networks are constrained via their architecture to focus on relations between perceptual inputs, rather than the attributes of individual inputs.
We review a family of models that employ this approach to induce abstractions in a data-efficient manner.
arXiv Detail & Related papers (2023-09-12T22:44:14Z) - From Patches to Objects: Exploiting Spatial Reasoning for Better Visual
Representations [2.363388546004777]
We propose a novel auxiliary pretraining method that is based on spatial reasoning.
Our proposed method takes advantage of a more flexible formulation of contrastive learning by introducing spatial reasoning as an auxiliary task for discriminative self-supervised methods.
arXiv Detail & Related papers (2023-05-21T07:46:46Z) - Contextualization and Generalization in Entity and Relation Extraction [0.0]
We study the behaviour of state-of-the-art models regarding generalization to facts unseen during training.
Traditional benchmarks present important lexical overlap between mentions and relations used for training and evaluating models.
We propose empirical studies to separate performance based on mention and relation overlap with the training set.
arXiv Detail & Related papers (2022-06-15T14:16:42Z) - Causal Reasoning Meets Visual Representation Learning: A Prospective
Study [117.08431221482638]
Lack of interpretability, robustness, and out-of-distribution generalization are becoming the challenges of the existing visual models.
Inspired by the strong inference ability of human-level agents, recent years have witnessed great effort in developing causal reasoning paradigms.
This paper aims to provide a comprehensive overview of this emerging field, attract attention, encourage discussions, bring to the forefront the urgency of developing novel causal reasoning methods.
arXiv Detail & Related papers (2022-04-26T02:22:28Z) - On Generalizing Beyond Domains in Cross-Domain Continual Learning [91.56748415975683]
Deep neural networks often suffer from catastrophic forgetting of previously learned knowledge after learning a new task.
Our proposed approach learns new tasks under domain shift with accuracy boosts up to 10% on challenging datasets such as DomainNet and OfficeHome.
arXiv Detail & Related papers (2022-03-08T09:57:48Z) - Measuring disentangled generative spatio-temporal representation [9.264758623908813]
We adopt two state-the-art disentangled representation learning methods and apply them to three large-scale public-temporal datasets.
We find that our methods can be used to discover real-world-world semantics to describe the variables in the learned representation.
arXiv Detail & Related papers (2022-02-10T03:57:06Z) - Learning a Domain-Agnostic Visual Representation for Autonomous Driving
via Contrastive Loss [25.798361683744684]
Domain-Agnostic Contrastive Learning (DACL) is a two-stage unsupervised domain adaptation framework with cyclic adversarial training and contrastive loss.
Our proposed approach achieves better performance in the monocular depth estimation task compared to previous state-of-the-art methods.
arXiv Detail & Related papers (2021-03-10T07:06:03Z) - Relation-Guided Representation Learning [53.60351496449232]
We propose a new representation learning method that explicitly models and leverages sample relations.
Our framework well preserves the relations between samples.
By seeking to embed samples into subspace, we show that our method can address the large-scale and out-of-sample problem.
arXiv Detail & Related papers (2020-07-11T10:57:45Z) - Learning What Makes a Difference from Counterfactual Examples and
Gradient Supervision [57.14468881854616]
We propose an auxiliary training objective that improves the generalization capabilities of neural networks.
We use pairs of minimally-different examples with different labels, a.k.a counterfactual or contrasting examples, which provide a signal indicative of the underlying causal structure of the task.
Models trained with this technique demonstrate improved performance on out-of-distribution test sets.
arXiv Detail & Related papers (2020-04-20T02:47:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.