Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex

• URL: http://arxiv.org/abs/2406.16935v1
• Date: Sun, 16 Jun 2024 20:33:57 GMT
• Title: Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex
• Authors: Spandan Madan, Will Xiao, Mingran Cao, Hanspeter Pfister, Margaret Livingstone, Gabriel Kreiman,
• Abstract summary: textitMacaqueITBench is a large-scale dataset of neural population responses from the macaque inferior temporal (IT) cortex. We investigated the impact of distribution shifts on models predicting neural activity by dividing the images into Out-Of-Distribution (OOD) train and test splits.
• Score: 26.91313901714098
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We characterized the generalization capabilities of DNN-based encoding models when predicting neuronal responses from the visual cortex. We collected \textit{MacaqueITBench}, a large-scale dataset of neural population responses from the macaque inferior temporal (IT) cortex to over $300,000$ images, comprising $8,233$ unique natural images presented to seven monkeys over $109$ sessions. Using \textit{MacaqueITBench}, we investigated the impact of distribution shifts on models predicting neural activity by dividing the images into Out-Of-Distribution (OOD) train and test splits. The OOD splits included several different image-computable types including image contrast, hue, intensity, temperature, and saturation. Compared to the performance on in-distribution test images -- the conventional way these models have been evaluated -- models performed worse at predicting neuronal responses to out-of-distribution images, retaining as little as $20\%$ of the performance on in-distribution test images. The generalization performance under OOD shifts can be well accounted by a simple image similarity metric -- the cosine distance between image representations extracted from a pre-trained object recognition model is a strong predictor of neural predictivity under different distribution shifts. The dataset of images, neuronal firing rate recordings, and computational benchmarks are hosted publicly at: https://bit.ly/3zeutVd.

Related papers

• Bayesian Inverse Graphics for Few-Shot Concept Learning [3.475273727432576]
  We present a Bayesian model of perception that learns using only minimal data. We show how this representation can be used for downstream tasks such as few-shot classification and estimation.
  arXiv  Detail & Related papers  (2024-09-12T18:30:41Z)
• Continuous time recurrent neural networks: overview and application to forecasting blood glucose in the intensive care unit [56.801856519460465]
  Continuous time autoregressive recurrent neural networks (CTRNNs) are a deep learning model that account for irregular observations. We demonstrate the application of these models to probabilistic forecasting of blood glucose in a critical care setting.
  arXiv  Detail & Related papers  (2023-04-14T09:39:06Z)
• Masked Images Are Counterfactual Samples for Robust Fine-tuning [77.82348472169335]
  Fine-tuning deep learning models can lead to a trade-off between in-distribution (ID) performance and out-of-distribution (OOD) robustness. We propose a novel fine-tuning method, which uses masked images as counterfactual samples that help improve the robustness of the fine-tuning model.
  arXiv  Detail & Related papers  (2023-03-06T11:51:28Z)
• Adversarial Sampling for Fairness Testing in Deep Neural Network [0.0]
  adversarial sampling to test for fairness in prediction of deep neural network model across different classes of image in a given dataset. We trained our neural network model on the original image, and without training our model on the perturbed or attacked image. When we feed the adversarial samplings to our model, it was able to predict the original category/ class of the image the adversarial sample belongs to.
  arXiv  Detail & Related papers  (2023-03-06T03:55:37Z)
• Convolutional Neural Generative Coding: Scaling Predictive Coding to Natural Images [79.07468367923619]
  We develop convolutional neural generative coding (Conv-NGC) We implement a flexible neurobiologically-motivated algorithm that progressively refines latent state maps. We study the effectiveness of our brain-inspired neural system on the tasks of reconstruction and image denoising.
  arXiv  Detail & Related papers  (2022-11-22T06:42:41Z)
• Decoupled Mixup for Generalized Visual Recognition [71.13734761715472]
  We propose a novel "Decoupled-Mixup" method to train CNN models for visual recognition. Our method decouples each image into discriminative and noise-prone regions, and then heterogeneously combines these regions to train CNN models. Experiment results show the high generalization performance of our method on testing data that are composed of unseen contexts.
  arXiv  Detail & Related papers  (2022-10-26T15:21:39Z)
• Corrupted Image Modeling for Self-Supervised Visual Pre-Training [103.99311611776697]
  We introduce Corrupted Image Modeling (CIM) for self-supervised visual pre-training. CIM uses an auxiliary generator with a small trainable BEiT to corrupt the input image instead of using artificial mask tokens. After pre-training, the enhancer can be used as a high-capacity visual encoder for downstream tasks.
  arXiv  Detail & Related papers  (2022-02-07T17:59:04Z)
• A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributes [58.633364000258645]
  We call this dataset RIVAL10 consisting of roughly $26k$ instances over $10$ classes. We evaluate the sensitivity of a broad set of models to noise corruptions in foregrounds, backgrounds and attributes. In our analysis, we consider diverse state-of-the-art architectures (ResNets, Transformers) and training procedures (CLIP, SimCLR, DeiT, Adversarial Training)
  arXiv  Detail & Related papers  (2022-01-26T06:31:28Z)
• Investigation of REFINED CNN ensemble learning for anti-cancer drug sensitivity prediction [0.0]
  Anti-cancer drug sensitivity prediction using deep learning models for individual cell line is a significant challenge in personalized medicine. REFINED CNN (Convolutional Neural Network) based models have shown promising results in drug sensitivity prediction. We consider predictions based on ensembles built from such mappings that can improve upon the best single REFINED CNN model prediction.
  arXiv  Detail & Related papers  (2020-09-09T02:27:29Z)
• Improving Calibration and Out-of-Distribution Detection in Medical Image Segmentation with Convolutional Neural Networks [8.219843232619551]
  Convolutional Neural Networks (CNNs) have shown to be powerful medical image segmentation models. We advocate for multi-task learning, i.e., training a single model on several different datasets. We show that not only a single CNN learns to automatically recognize the context and accurately segment the organ of interest in each context, but also that such a joint model often has more accurate and better-calibrated predictions.
  arXiv  Detail & Related papers  (2020-04-12T23:42:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.