Computer Vision Models Show Human-Like Sensitivity to Geometric and Topological Concepts

• URL: http://arxiv.org/abs/2505.13281v1
• Date: Mon, 19 May 2025 16:04:53 GMT
• Title: Computer Vision Models Show Human-Like Sensitivity to Geometric and Topological Concepts
• Authors: Zekun Wang, Sashank Varma,
• Abstract summary: We investigate computer vision models and human sensitivity to geometric and topological (GT) concepts.<n>We do so using computer visions models, which are trained on large image datasets.<n> Transformer-based models achieve the highest overall accuracy, surpassing that of young children.
• Score: 1.935452308279137
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the rapid improvement of machine learning (ML) models, cognitive scientists are increasingly asking about their alignment with how humans think. Here, we ask this question for computer vision models and human sensitivity to geometric and topological (GT) concepts. Under the core knowledge account, these concepts are innate and supported by dedicated neural circuitry. In this work, we investigate an alternative explanation, that GT concepts are learned ``for free'' through everyday interaction with the environment. We do so using computer visions models, which are trained on large image datasets. We build on prior studies to investigate the overall performance and human alignment of three classes of models -- convolutional neural networks (CNNs), transformer-based models, and vision-language models -- on an odd-one-out task testing 43 GT concepts spanning seven classes. Transformer-based models achieve the highest overall accuracy, surpassing that of young children. They also show strong alignment with children's performance, finding the same classes of concepts easy vs. difficult. By contrast, vision-language models underperform their vision-only counterparts and deviate further from human profiles, indicating that na\"ive multimodality might compromise abstract geometric sensitivity. These findings support the use of computer vision models to evaluate the sufficiency of the learning account for explaining human sensitivity to GT concepts, while also suggesting that integrating linguistic and visual representations might have unpredicted deleterious consequences.

Related papers

• Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models [51.900488744931785]
  We introduce the Visual Graph Arena (VGA) to evaluate and improve AI systems' capacity for visual abstraction.<n>Humans achieve near-perfect accuracy across tasks, while models totally failed on isomorphism detection and showed limited success in path/cycle tasks.<n>By isolating the challenge of representation-invariant reasoning, the VGA provides a framework to drive progress toward human-like conceptualization in AI visual models.
  arXiv  Detail & Related papers  (2025-06-06T17:06:25Z)
• Testing the limits of fine-tuning to improve reasoning in vision language models [51.58859621164201]
  We introduce visual stimuli and human judgments on visual cognition tasks to evaluate performance across cognitive domains.<n>We fine-tune models on ground truth data for intuitive physics and causal reasoning.<n>We find that fine-tuning does not contribute to robust human-like generalization to data with other visual characteristics or to tasks in other cognitive domains.
  arXiv  Detail & Related papers  (2025-02-21T18:58:30Z)
• Autoregressive Models in Vision: A Survey [119.23742136065307]
  This survey comprehensively examines the literature on autoregressive models applied to vision. We divide visual autoregressive models into three general sub-categories, including pixel-based, token-based, and scale-based models. We present a multi-faceted categorization of autoregressive models in computer vision, including image generation, video generation, 3D generation, and multi-modal generation.
  arXiv  Detail & Related papers  (2024-11-08T17:15:12Z)
• When Does Perceptual Alignment Benefit Vision Representations? [76.32336818860965]
  We investigate how aligning vision model representations to human perceptual judgments impacts their usability. We find that aligning models to perceptual judgments yields representations that improve upon the original backbones across many downstream tasks. Our results suggest that injecting an inductive bias about human perceptual knowledge into vision models can contribute to better representations.
  arXiv  Detail & Related papers  (2024-10-14T17:59:58Z)
• Aligning Machine and Human Visual Representations across Abstraction Levels [42.86478924838503]
  Deep neural networks have achieved success across a wide range of applications, including as models of human behavior in vision tasks. However, neural network training and human learning differ in fundamental ways, and neural networks often fail to generalize as robustly as humans do. We highlight a key misalignment between vision models and humans: whereas human conceptual knowledge is hierarchically organized from fine- to coarse-scale distinctions, model representations do not accurately capture all these levels of abstraction. To address this misalignment, we first train a teacher model to imitate human judgments, then transfer human-like structure from its representations into pretrained state-of-the
  arXiv  Detail & Related papers  (2024-09-10T13:41:08Z)
• How Well Do Deep Learning Models Capture Human Concepts? The Case of the Typicality Effect [2.3622884172290255]
  Recent research looking for human-like typicality effects in language and vision models has focused on models of a single modality. This study expands this behavioral evaluation of models by considering a broader range of language and vision models. It also evaluates whether the combined typicality predictions of vision + language model pairs, as well as a multimodal CLIP-based model, are better aligned with human typicality judgments than those of models of either modality alone.
  arXiv  Detail & Related papers  (2024-05-25T08:38:30Z)
• Visual Grounding Helps Learn Word Meanings in Low-Data Regimes [47.7950860342515]
  Modern neural language models (LMs) are powerful tools for modeling human sentence production and comprehension. But to achieve these results, LMs must be trained in distinctly un-human-like ways. Do models trained more naturalistically -- with grounded supervision -- exhibit more humanlike language learning? We investigate this question in the context of word learning, a key sub-task in language acquisition.
  arXiv  Detail & Related papers  (2023-10-20T03:33:36Z)
• Evaluating alignment between humans and neural network representations in image-based learning tasks [5.657101730705275]
  We tested how well the representations of $86$ pretrained neural network models mapped to human learning trajectories.<n>We found that while training dataset size was a core determinant of alignment with human choices, contrastive training with multi-modal data (text and imagery) was a common feature of currently publicly available models that predicted human generalisation.<n>In conclusion, pretrained neural networks can serve to extract representations for cognitive models, as they appear to capture some fundamental aspects of cognition that are transferable across tasks.
  arXiv  Detail & Related papers  (2023-06-15T08:18:29Z)
• Degraded Polygons Raise Fundamental Questions of Neural Network Perception [5.423100066629618]
  We revisit the task of recovering images under degradation, first introduced over 30 years ago in the Recognition-by-Components theory of human vision. We implement the Automated Shape Recoverability Test for rapidly generating large-scale datasets of perimeter-degraded regular polygons. We find that neural networks' behavior on this simple task conflicts with human behavior.
  arXiv  Detail & Related papers  (2023-06-08T06:02:39Z)
• Human alignment of neural network representations [28.32452075196472]
  We investigate the factors that affect the alignment between the representations learned by neural networks and human mental representations inferred from behavioral responses.<n>We find that model scale and architecture have essentially no effect on the alignment with human behavioral responses.<n>We find that some human concepts such as food and animals are well-represented by neural networks whereas others such as royal or sports-related objects are not.
  arXiv  Detail & Related papers  (2022-11-02T15:23:16Z)
• 3D Neural Scene Representations for Visuomotor Control [78.79583457239836]
  We learn models for dynamic 3D scenes purely from 2D visual observations. A dynamics model, constructed over the learned representation space, enables visuomotor control for challenging manipulation tasks.
  arXiv  Detail & Related papers  (2021-07-08T17:49:37Z)
• Deep Reinforcement Learning Models Predict Visual Responses in the Brain: A Preliminary Result [1.0323063834827415]
  We use reinforcement learning to train neural network models to play a 3D computer game. We find that these reinforcement learning models achieve neural response prediction accuracy scores in the early visual areas. In contrast, the supervised neural network models yield better neural response predictions in the higher visual areas.
  arXiv  Detail & Related papers  (2021-06-18T13:10:06Z)
• Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning [78.13740873213223]
  Bongard problems (BPs) were introduced as an inspirational challenge for visual cognition in intelligent systems. We propose a new benchmark Bongard-LOGO for human-level concept learning and reasoning.
  arXiv  Detail & Related papers  (2020-10-02T03:19:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.