Extrapolation Frameworks in Cognitive Psychology Suitable for Study of Image Classification Models

• URL: http://arxiv.org/abs/2112.03411v1
• Date: Mon, 6 Dec 2021 23:06:31 GMT
• Title: Extrapolation Frameworks in Cognitive Psychology Suitable for Study of Image Classification Models
• Authors: Roozbeh Yousefzadeh, Jessica A. Mollick
• Abstract summary: In contrast to the deep learning literature, in cognitive science, psychology, and neuroscience, extrapolation and learning are often studied in tandem. We propose a novel extrapolation framework for the mathematical study of deep learning models.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the functional task of deep learning image classification models and show that image classification requires extrapolation capabilities. This suggests that new theories have to be developed for the understanding of deep learning as the current theory assumes models are solely interpolating, leaving many questions about them unanswered. We investigate the pixel space and also the feature spaces extracted from images by trained models (in their hidden layers, including the 64-dimensional feature space in the last hidden layer of pre-trained residual neural networks), and also the feature space extracted by wavelets/shearlets. In all these domains, testing samples considerably fall outside the convex hull of training sets, and image classification requires extrapolation. In contrast to the deep learning literature, in cognitive science, psychology, and neuroscience, extrapolation and learning are often studied in tandem. Moreover, many aspects of human visual cognition and behavior are reported to involve extrapolation. We propose a novel extrapolation framework for the mathematical study of deep learning models. In our framework, we use the term extrapolation in this specific way of extrapolating outside the convex hull of training set (in the pixel space or feature space) but within the specific scope defined by the training data, the same way extrapolation is defined in many studies in cognitive science. We explain that our extrapolation framework can provide novel answers to open research problems about deep learning including their over-parameterization, their training regime, out-of-distribution detection, etc. We also see that the extent of extrapolation is negligible in learning tasks where deep learning is reported to have no advantage over simple models.

Related papers

• Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond [61.18736646013446]
  In pursuit of a deeper understanding of its surprising behaviors, we investigate the utility of a simple yet accurate model of a trained neural network. Across three case studies, we illustrate how it can be applied to derive new empirical insights on a diverse range of prominent phenomena.
  arXiv  Detail & Related papers  (2024-10-31T22:54:34Z)
• How language models extrapolate outside the training data: A case study in Textualized Gridworld [32.5268320198854]
  We show that conventional approaches, including next token prediction and Chain of Thought finetuning, fail to extrapolate in larger, unseen environments. We propose cognitive maps for path planning, a novel CoT framework that simulates humanlike mental representations. Our finding that these cognitive maps require specialized training schemes opens up important questions about developing general-purpose cognitive maps in language models.
  arXiv  Detail & Related papers  (2024-06-21T16:10:05Z)
• Free-ATM: Exploring Unsupervised Learning on Diffusion-Generated Images with Free Attention Masks [64.67735676127208]
  Text-to-image diffusion models have shown great potential for benefiting image recognition. Although promising, there has been inadequate exploration dedicated to unsupervised learning on diffusion-generated images. We introduce customized solutions by fully exploiting the aforementioned free attention masks.
  arXiv  Detail & Related papers  (2023-08-13T10:07:46Z)
• Does Deep Learning Learn to Abstract? A Systematic Probing Framework [69.2366890742283]
  Abstraction is a desirable capability for deep learning models, which means to induce abstract concepts from concrete instances and flexibly apply them beyond the learning context. We introduce a systematic probing framework to explore the abstraction capability of deep learning models from a transferability perspective.
  arXiv  Detail & Related papers  (2023-02-23T12:50:02Z)
• Deep networks for system identification: a Survey [56.34005280792013]
  System identification learns mathematical descriptions of dynamic systems from input-output data. Main aim of the identified model is to predict new data from previous observations. We discuss architectures commonly adopted in the literature, like feedforward, convolutional, and recurrent networks.
  arXiv  Detail & Related papers  (2023-01-30T12:38:31Z)
• Reliable extrapolation of deep neural operators informed by physics or sparse observations [2.887258133992338]
  Deep neural operators can learn nonlinear mappings between infinite-dimensional function spaces via deep neural networks. DeepONets provide a new simulation paradigm in science and engineering. We propose five reliable learning methods that guarantee a safe prediction under extrapolation.
  arXiv  Detail & Related papers  (2022-12-13T03:02:46Z)
• Dynamic Latent Separation for Deep Learning [67.62190501599176]
  A core problem in machine learning is to learn expressive latent variables for model prediction on complex data. Here, we develop an approach that improves expressiveness, provides partial interpretation, and is not restricted to specific applications.
  arXiv  Detail & Related papers  (2022-10-07T17:56:53Z)
• Decision boundaries and convex hulls in the feature space that deep learning functions learn from images [0.0]
  We study the properties of a low-dimensional manifold that models extract and learn from images. For image classification models, their last hidden layer is the one where images of each class is separated from other classes and it also has the least number of features. We observe that geometric arrangements of decision boundaries in feature space is significantly different compared to pixel space.
  arXiv  Detail & Related papers  (2022-02-05T15:09:51Z)
• Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach [80.8446673089281]
  We propose a new learning paradigm with graph representation and learning. Our framework contains two modules: 1) a backbone network (e.g., feedforward neural nets) as a lower model takes features as input and outputs predicted labels; 2) a graph neural network as an upper model learns to extrapolate embeddings for new features via message passing over a feature-data graph built from observed data.
  arXiv  Detail & Related papers  (2021-10-09T09:02:45Z)
• Extracting Global Dynamics of Loss Landscape in Deep Learning Models [0.0]
  We present a toolkit for the Dynamical Organization Of Deep Learning Loss Landscapes, or DOODL3. DOODL3 formulates the training of neural networks as a dynamical system, analyzes the learning process, and presents an interpretable global view of trajectories in the loss landscape.
  arXiv  Detail & Related papers  (2021-06-14T18:07:05Z)
• Interactive Visual Study of Multiple Attributes Learning Model of X-Ray Scattering Images [34.95218692917125]
  We present an interactive system for domain scientists to visually study the multiple attributes learning models applied to x-ray scattering images. The exploration is guided by the manifestation of model performance related to mutual relationships among attributes. The system thus supports domain scientists to improve the training dataset and model, find questionable attributes labels, and identify outlier images or spurious data clusters.
  arXiv  Detail & Related papers  (2020-09-03T00:38:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.