Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex

• URL: http://arxiv.org/abs/2306.03779v1
• Date: Tue, 6 Jun 2023 15:34:45 GMT
• Title: Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex
• Authors: Drew Linsley, Ivan F. Rodriguez, Thomas Fel, Michael Arcaro, Saloni Sharma, Margaret Livingstone, Thomas Serre
• Abstract summary: We show that object recognition accuracy of deep neural networks (DNNs) correlates with their ability to predict neural responses to natural images in the inferotemporal (IT) cortex. Our results suggest that harmonized DNNs break the trade-off between ImageNet accuracy and neural prediction accuracy.
• Score: 8.45100792118802
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: One of the most impactful findings in computational neuroscience over the past decade is that the object recognition accuracy of deep neural networks (DNNs) correlates with their ability to predict neural responses to natural images in the inferotemporal (IT) cortex. This discovery supported the long-held theory that object recognition is a core objective of the visual cortex, and suggested that more accurate DNNs would serve as better models of IT neuron responses to images. Since then, deep learning has undergone a revolution of scale: billion parameter-scale DNNs trained on billions of images are rivaling or outperforming humans at visual tasks including object recognition. Have today's DNNs become more accurate at predicting IT neuron responses to images as they have grown more accurate at object recognition? Surprisingly, across three independent experiments, we find this is not the case. DNNs have become progressively worse models of IT as their accuracy has increased on ImageNet. To understand why DNNs experience this trade-off and evaluate if they are still an appropriate paradigm for modeling the visual system, we turn to recordings of IT that capture spatially resolved maps of neuronal activity elicited by natural images. These neuronal activity maps reveal that DNNs trained on ImageNet learn to rely on different visual features than those encoded by IT and that this problem worsens as their accuracy increases. We successfully resolved this issue with the neural harmonizer, a plug-and-play training routine for DNNs that aligns their learned representations with humans. Our results suggest that harmonized DNNs break the trade-off between ImageNet accuracy and neural prediction accuracy that assails current DNNs and offer a path to more accurate models of biological vision.

Related papers

• When Spiking neural networks meet temporal attention image decoding and adaptive spiking neuron [7.478056407323783]
  Spiking Neural Networks (SNNs) are capable of encoding and processing temporal information in a biologically plausible way. We propose a novel method for image decoding based on temporal attention (TAID) and an adaptive Leaky-Integrate-and-Fire neuron model.
  arXiv  Detail & Related papers  (2024-06-05T08:21:55Z)
• Are Deep Neural Networks Adequate Behavioural Models of Human Visual Perception? [8.370048099732573]
  Deep neural networks (DNNs) are machine learning algorithms that have revolutionised computer vision. We argue that it is important to distinguish between statistical tools and computational models. We dispel a number of myths surrounding DNNs in vision science.
  arXiv  Detail & Related papers  (2023-05-26T15:31:06Z)
• A Hybrid Neural Coding Approach for Pattern Recognition with Spiking Neural Networks [53.31941519245432]
  Brain-inspired spiking neural networks (SNNs) have demonstrated promising capabilities in solving pattern recognition tasks. These SNNs are grounded on homogeneous neurons that utilize a uniform neural coding for information representation. In this study, we argue that SNN architectures should be holistically designed to incorporate heterogeneous coding schemes.
  arXiv  Detail & Related papers  (2023-05-26T02:52:12Z)
• Models Developed for Spiking Neural Networks [0.5801044612920815]
  Spiking neural networks (SNNs) have been around for a long time, and they have been investigated to understand the dynamics of the brain. In this work, we reviewed the structures and performances of SNNs on image classification tasks. The comparisons illustrate that these networks show great capabilities for more complicated problems.
  arXiv  Detail & Related papers  (2022-12-08T16:18:53Z)
• Harmonizing the object recognition strategies of deep neural networks with humans [10.495114898741205]
  We show that state-of-the-art deep neural networks (DNNs) are becoming less aligned with humans as their accuracy improves. Our work represents the first demonstration that the scaling laws that are guiding the design of DNNs today have also produced worse models of human vision.
  arXiv  Detail & Related papers  (2022-11-08T20:03:49Z)
• Adapting Brain-Like Neural Networks for Modeling Cortical Visual Prostheses [68.96380145211093]
  Cortical prostheses are devices implanted in the visual cortex that attempt to restore lost vision by electrically stimulating neurons. Currently, the vision provided by these devices is limited, and accurately predicting the visual percepts resulting from stimulation is an open challenge. We propose to address this challenge by utilizing 'brain-like' convolutional neural networks (CNNs), which have emerged as promising models of the visual system.
  arXiv  Detail & Related papers  (2022-09-27T17:33:19Z)
• Deep Reinforcement Learning Guided Graph Neural Networks for Brain Network Analysis [61.53545734991802]
  We propose a novel brain network representation framework, namely BN-GNN, which searches for the optimal GNN architecture for each brain network. Our proposed BN-GNN improves the performance of traditional GNNs on different brain network analysis tasks.
  arXiv  Detail & Related papers  (2022-03-18T07:05:27Z)
• Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks [82.54695985117783]
  We investigate the suitability of state-of-the-art deep learning architectures for continuous emotion recognition using long video sequences captured in-the-wild. We have developed and evaluated convolutional recurrent neural networks combining 2D-CNNs and long short term-memory units, and inflated 3D-CNN models, which are built by inflating the weights of a pre-trained 2D-CNN model during fine-tuning.
  arXiv  Detail & Related papers  (2020-11-18T13:42:05Z)
• Boosting Deep Neural Networks with Geometrical Prior Knowledge: A Survey [77.99182201815763]
  Deep Neural Networks (DNNs) achieve state-of-the-art results in many different problem settings. DNNs are often treated as black box systems, which complicates their evaluation and validation. One promising field, inspired by the success of convolutional neural networks (CNNs) in computer vision tasks, is to incorporate knowledge about symmetric geometrical transformations.
  arXiv  Detail & Related papers  (2020-06-30T14:56:05Z)
• High-contrast "gaudy" images improve the training of deep neural network models of visual cortex [21.219431687928523]
  A key challenge in understanding the sensory transformations of the visual system is to obtain a highly predictive model of responses from visual cortical neurons. Deep neural networks (DNNs) provide a promising candidate for such a model. We propose images that train highly-predictive DNNs with as little training data as possible.
  arXiv  Detail & Related papers  (2020-06-13T20:05:16Z)
• Neural Additive Models: Interpretable Machine Learning with Neural Nets [77.66871378302774]
  Deep neural networks (DNNs) are powerful black-box predictors that have achieved impressive performance on a wide variety of tasks. We propose Neural Additive Models (NAMs) which combine some of the expressivity of DNNs with the inherent intelligibility of generalized additive models. NAMs learn a linear combination of neural networks that each attend to a single input feature.
  arXiv  Detail & Related papers  (2020-04-29T01:28:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.