How Well Do Deep Learning Models Capture Human Concepts? The Case of the Typicality Effect

• URL: http://arxiv.org/abs/2405.16128v1
• Date: Sat, 25 May 2024 08:38:30 GMT
• Title: How Well Do Deep Learning Models Capture Human Concepts? The Case of the Typicality Effect
• Authors: Siddhartha K. Vemuri, Raj Sanjay Shah, Sashank Varma,
• Abstract summary: Recent research looking for human-like typicality effects in language and vision models has focused on models of a single modality. This study expands this behavioral evaluation of models by considering a broader range of language and vision models. It also evaluates whether the combined typicality predictions of vision + language model pairs, as well as a multimodal CLIP-based model, are better aligned with human typicality judgments than those of models of either modality alone.
• Score: 2.3622884172290255
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: How well do representations learned by ML models align with those of humans? Here, we consider concept representations learned by deep learning models and evaluate whether they show a fundamental behavioral signature of human concepts, the typicality effect. This is the finding that people judge some instances (e.g., robin) of a category (e.g., Bird) to be more typical than others (e.g., penguin). Recent research looking for human-like typicality effects in language and vision models has focused on models of a single modality, tested only a small number of concepts, and found only modest correlations with human typicality ratings. The current study expands this behavioral evaluation of models by considering a broader range of language (N = 8) and vision (N = 10) model architectures. It also evaluates whether the combined typicality predictions of vision + language model pairs, as well as a multimodal CLIP-based model, are better aligned with human typicality judgments than those of models of either modality alone. Finally, it evaluates the models across a broader range of concepts (N = 27) than prior studies. There were three important findings. First, language models better align with human typicality judgments than vision models. Second, combined language and vision models (e.g., AlexNet + MiniLM) better predict the human typicality data than the best-performing language model (i.e., MiniLM) or vision model (i.e., ViT-Huge) alone. Third, multimodal models (i.e., CLIP ViT) show promise for explaining human typicality judgments. These results advance the state-of-the-art in aligning the conceptual representations of ML models and humans. A methodological contribution is the creation of a new image set for testing the conceptual alignment of vision models.

Related papers

• Human-Centric Foundation Models: Perception, Generation and Agentic Modeling [79.97999901785772]
  Human-centric Foundation Models unify diverse human-centric tasks into a single framework. We present a comprehensive overview of HcFMs by proposing a taxonomy that categorizes current approaches into four groups. This survey aims to serve as a roadmap for researchers and practitioners working towards more robust, versatile, and intelligent digital human and embodiments modeling.
  arXiv  Detail & Related papers  (2025-02-12T16:38:40Z)
• Autoregressive Models in Vision: A Survey [119.23742136065307]
  This survey comprehensively examines the literature on autoregressive models applied to vision. We divide visual autoregressive models into three general sub-categories, including pixel-based, token-based, and scale-based models. We present a multi-faceted categorization of autoregressive models in computer vision, including image generation, video generation, 3D generation, and multi-modal generation.
  arXiv  Detail & Related papers  (2024-11-08T17:15:12Z)
• Evaluating Multiview Object Consistency in Humans and Image Models [68.36073530804296]
  We leverage an experimental design from the cognitive sciences which requires zero-shot visual inferences about object shape. We collect 35K trials of behavioral data from over 500 participants. We then evaluate the performance of common vision models.
  arXiv  Detail & Related papers  (2024-09-09T17:59:13Z)
• DevBench: A multimodal developmental benchmark for language learning [0.34129029452670606]
  We introduce DevBench, a benchmark for evaluating vision-language models on tasks and behavioral data. We show that DevBench provides a benchmark for comparing models to human language development. These comparisons highlight ways in which model and human language learning processes diverge.
  arXiv  Detail & Related papers  (2024-06-14T17:49:41Z)
• Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms [91.19304518033144]
  We aim to align vision models with human aesthetic standards in a retrieval system. We propose a preference-based reinforcement learning method that fine-tunes the vision models to better align the vision models with human aesthetics.
  arXiv  Detail & Related papers  (2024-06-13T17:59:20Z)
• Longer Fixations, More Computation: Gaze-Guided Recurrent Neural Networks [12.57650361978445]
  Humans read texts at a varying pace, while machine learning models treat each token in the same way. In this paper, we convert this intuition into a set of novel models with fixation-guided parallel RNNs or layers. We find that, interestingly, the fixation duration predicted by neural networks bears some resemblance to humans' fixation.
  arXiv  Detail & Related papers  (2023-10-31T21:32:11Z)
• Turning large language models into cognitive models [0.0]
  We show that large language models can be turned into cognitive models. These models offer accurate representations of human behavior, even outperforming traditional cognitive models in two decision-making domains. Taken together, these results suggest that large, pre-trained models can be adapted to become generalist cognitive models.
  arXiv  Detail & Related papers  (2023-06-06T18:00:01Z)
• Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models [57.08925810659545]
  We conduct a comparative analysis of the visual representations in existing vision-and-language models and vision-only models. Our empirical observations suggest that vision-and-language models are better at label prediction tasks. We hope our study sheds light on the role of language in visual learning, and serves as an empirical guide for various pretrained models.
  arXiv  Detail & Related papers  (2022-12-01T05:00:18Z)
• UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes [91.24112204588353]
  We introduce UViM, a unified approach capable of modeling a wide range of computer vision tasks. In contrast to previous models, UViM has the same functional form for all tasks. We demonstrate the effectiveness of UViM on three diverse and challenging vision tasks.
  arXiv  Detail & Related papers  (2022-05-20T17:47:59Z)
• Exploring Alignment of Representations with Human Perception [47.53970721813083]
  We show that inputs that are mapped to similar representations by the model should be perceived similarly by humans. Our approach yields a measure of the extent to which a model is aligned with human perception. We find that various properties of a model like its architecture, training paradigm, training loss, and data augmentation play a significant role in learning representations that are aligned with human perception.
  arXiv  Detail & Related papers  (2021-11-29T17:26:50Z)
• End-to-end Deep Prototype and Exemplar Models for Predicting Human Behavior [10.361297404586033]
  We extend classic prototype and exemplar models to learn both stimulus and category representations jointly from raw input. This new class of models can be parameterized by deep neural networks (DNN) and trained end-to-end. Compared to typical DNNs, we find that their cognitively inspired counterparts both provide better intrinsic fit to human behavior and improve ground-truth classification.
  arXiv  Detail & Related papers  (2020-07-17T02:32:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.