Visual Anagrams Reveal Hidden Differences in Holistic Shape Processing Across Vision Models

• URL: http://arxiv.org/abs/2507.00493v2
• Date: Thu, 03 Jul 2025 18:51:47 GMT
• Title: Visual Anagrams Reveal Hidden Differences in Holistic Shape Processing Across Vision Models
• Authors: Fenil R. Doshi, Thomas Fel, Talia Konkle, George Alvarez,
• Abstract summary: We recast shape evaluation as a matter of absolute configural competence, operationalized by the Configural Shape Score (CSS)<n>CSS uncovers a broad spectrum of configural sensitivity with fully self-supervised and language-aligned transformers.<n>We propose that the path toward truly robust, generalizable, and human-like vision systems may not lie in forcing an artificial choice between shape and texture.
• Score: 4.749824105387293
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans are able to recognize objects based on both local texture cues and the configuration of object parts, yet contemporary vision models primarily harvest local texture cues, yielding brittle, non-compositional features. Work on shape-vs-texture bias has pitted shape and texture representations in opposition, measuring shape relative to texture, ignoring the possibility that models (and humans) can simultaneously rely on both types of cues, and obscuring the absolute quality of both types of representation. We therefore recast shape evaluation as a matter of absolute configural competence, operationalized by the Configural Shape Score (CSS), which (i) measures the ability to recognize both images in Object-Anagram pairs that preserve local texture while permuting global part arrangement to depict different object categories. Across 86 convolutional, transformer, and hybrid models, CSS (ii) uncovers a broad spectrum of configural sensitivity with fully self-supervised and language-aligned transformers -- exemplified by DINOv2, SigLIP2 and EVA-CLIP -- occupying the top end of the CSS spectrum. Mechanistic probes reveal that (iii) high-CSS networks depend on long-range interactions: radius-controlled attention masks abolish performance showing a distinctive U-shaped integration profile, and representational-similarity analyses expose a mid-depth transition from local to global coding. A BagNet control remains at chance (iv), ruling out "border-hacking" strategies. Finally, (v) we show that configural shape score also predicts other shape-dependent evals. Overall, we propose that the path toward truly robust, generalizable, and human-like vision systems may not lie in forcing an artificial choice between shape and texture, but rather in architectural and learning frameworks that seamlessly integrate both local-texture and global configural shape.

Related papers

• Dynamic Scene Understanding through Object-Centric Voxelization and Neural Rendering [57.895846642868904]
  We present a 3D generative model named DynaVol-S for dynamic scenes that enables object-centric learning.<n>voxelization infers per-object occupancy probabilities at individual spatial locations.<n>Our approach integrates 2D semantic features to create 3D semantic grids, representing the scene through multiple disentangled voxel grids.
  arXiv  Detail & Related papers  (2024-07-30T15:33:58Z)
• Boosting Cross-Domain Point Classification via Distilling Relational Priors from 2D Transformers [59.0181939916084]
  Traditional 3D networks mainly focus on local geometric details and ignore the topological structure between local geometries. We propose a novel Priors Distillation (RPD) method to extract priors from the well-trained transformers on massive images. Experiments on the PointDA-10 and the Sim-to-Real datasets verify that the proposed method consistently achieves the state-of-the-art performance of UDA for point cloud classification.
  arXiv  Detail & Related papers  (2024-07-26T06:29:09Z)
• Context and Geometry Aware Voxel Transformer for Semantic Scene Completion [7.147020285382786]
  Vision-based Semantic Scene Completion (SSC) has gained much attention due to its widespread applications in various 3D perception tasks. Existing sparse-to-dense approaches typically employ shared context-independent queries across various input images. We introduce a neural network named CGFormer to achieve semantic scene completion.
  arXiv  Detail & Related papers  (2024-05-22T14:16:30Z)
• N2F2: Hierarchical Scene Understanding with Nested Neural Feature Fields [112.02885337510716]
  Nested Neural Feature Fields (N2F2) is a novel approach that employs hierarchical supervision to learn a single feature field. We leverage a 2D class-agnostic segmentation model to provide semantically meaningful pixel groupings at arbitrary scales in the image space. Our approach outperforms the state-of-the-art feature field distillation methods on tasks such as open-vocabulary 3D segmentation and localization.
  arXiv  Detail & Related papers  (2024-03-16T18:50:44Z)
• Does resistance to style-transfer equal Global Shape Bias? Measuring network sensitivity to global shape configuration [6.047146237332764]
  Current benchmark for evaluating a model's global shape bias is a set of style-transferred images. We show that networks trained with style-transfer images indeed learn to ignore style, but its shape bias arises primarily from local detail.
  arXiv  Detail & Related papers  (2023-10-11T15:00:11Z)
• SVDFormer: Complementing Point Cloud via Self-view Augmentation and Self-structure Dual-generator [30.483163963846206]
  We propose a novel network, SVDFormer, to tackle two specific challenges in point cloud completion. We first design a Self-view Fusion Network that leverages multiple-view depth image information to observe incomplete self-shape. We then introduce a refinement module, called Self-structure Dual-generator, in which we incorporate learned shape priors and geometric self-similarities for producing new points.
  arXiv  Detail & Related papers  (2023-07-17T13:55:31Z)
• Flattening-Net: Deep Regular 2D Representation for 3D Point Cloud Analysis [66.49788145564004]
  We present an unsupervised deep neural architecture called Flattening-Net to represent irregular 3D point clouds of arbitrary geometry and topology. Our methods perform favorably against the current state-of-the-art competitors.
  arXiv  Detail & Related papers  (2022-12-17T15:05:25Z)
• Vision Transformers: From Semantic Segmentation to Dense Prediction [139.15562023284187]
  We explore the global context learning potentials of vision transformers (ViTs) for dense visual prediction. Our motivation is that through learning global context at full receptive field layer by layer, ViTs may capture stronger long-range dependency information. We formulate a family of Hierarchical Local-Global (HLG) Transformers, characterized by local attention within windows and global-attention across windows in a pyramidal architecture.
  arXiv  Detail & Related papers  (2022-07-19T15:49:35Z)
• Vision Transformer for NeRF-Based View Synthesis from a Single Input Image [49.956005709863355]
  We propose to leverage both the global and local features to form an expressive 3D representation. To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. Our method can render novel views from only a single input image and generalize across multiple object categories using a single model.
  arXiv  Detail & Related papers  (2022-07-12T17:52:04Z)
• Conformer: Local Features Coupling Global Representations for Visual Recognition [72.9550481476101]
  We propose a hybrid network structure, termed Conformer, to take advantage of convolutional operations and self-attention mechanisms for enhanced representation learning. Experiments show that Conformer, under the comparable parameter complexity, outperforms the visual transformer (DeiT-B) by 2.3% on ImageNet.
  arXiv  Detail & Related papers  (2021-05-09T10:00:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.