Human-like Object Grouping in Self-supervised Vision Transformers

• URL: http://arxiv.org/abs/2603.13994v1
• Date: Sat, 14 Mar 2026 15:43:10 GMT
• Title: Human-like Object Grouping in Self-supervised Vision Transformers
• Abstract summary: We introduce a behavioral benchmark in which participants make same/different object judgments for dot pairs on naturalistic scenes.<n>We test a diverse set of vision models using a simple readout from their representations to predict subjects' reaction times.<n>We show that self-supervised vision models capture object structure in a behaviorally human-like manner, and that Gram matrix structure plays a role in driving perceptual alignment.
• Score: 9.933177928703172
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision foundation models trained with self-supervised objectives achieve strong performance across diverse tasks and exhibit emergent object segmentation properties. However, their alignment with human object perception remains poorly understood. Here, we introduce a behavioral benchmark in which participants make same/different object judgments for dot pairs on naturalistic scenes, scaling up a classical psychophysics paradigm to over 1000 trials. We test a diverse set of vision models using a simple readout from their representations to predict subjects' reaction times. We observe a steady improvement across model generations, with both architecture and training objective contributing to alignment, and transformer-based models trained with the DINO self-supervised objective showing the strongest performance. To investigate the source of this improvement, we propose a novel metric to quantify the object-centric component of representations by measuring patch similarity within and between objects. Across models, stronger object-centric structure predicts human segmentation behavior more accurately. We further show that matching the Gram matrix of supervised transformer models, capturing similarity structure across image patches, with that of a self-supervised model through distillation improves their alignment with human behavior, converging with the prior finding that Gram anchoring improves DINOv3's feature quality. Together, these results demonstrate that self-supervised vision models capture object structure in a behaviorally human-like manner, and that Gram matrix structure plays a role in driving perceptual alignment.

Related papers

• Investigating Fine- and Coarse-grained Structural Correspondences Between Deep Neural Networks and Human Object Image Similarity Judgments Using Unsupervised Alignment [0.14999444543328289]
  We employ an unsupervised alignment method based on Gromov-Wasserstein Optimal Transport to compare human and model object representations.<n>We find that models trained with CLIP consistently achieve strong fine- and coarse-grained matching with human object representations.<n>Our results offer new insights into the role of linguistic information in acquiring precise object representations.
  arXiv  Detail & Related papers  (2025-05-22T09:06:06Z)
• A Data-Centric Revisit of Pre-Trained Vision Models for Robot Learning [67.72413262980272]
  Pre-trained vision models (PVMs) are fundamental to modern robotics, yet their optimal configuration remains unclear.<n>We develop SlotMIM, a method that induces object-centric representations by introducing a semantic bottleneck.<n>Our approach achieves significant improvements over prior work in image recognition, scene understanding, and robot learning evaluations.
  arXiv  Detail & Related papers  (2025-03-10T06:18:31Z)
• Zero-Shot Object-Centric Representation Learning [72.43369950684057]
  We study current object-centric methods through the lens of zero-shot generalization. We introduce a benchmark comprising eight different synthetic and real-world datasets. We find that training on diverse real-world images improves transferability to unseen scenarios.
  arXiv  Detail & Related papers  (2024-08-17T10:37:07Z)
• Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detection [116.21529970404653]
  We introduce SG2HOI+, a unified one-step model based on the Transformer architecture. Our approach employs two interactive hierarchical Transformers to seamlessly unify the tasks of SGG and HOI detection. Our approach achieves competitive performance when compared to state-of-the-art HOI methods.
  arXiv  Detail & Related papers  (2023-11-03T07:25:57Z)
• Affinity-based Attention in Self-supervised Transformers Predicts Dynamics of Object Grouping in Humans [2.485182034310303]
  We propose a model of human object-based attention spreading and segmentation. Our work provides new benchmarks for evaluating models of visual representation learning including Transformers.
  arXiv  Detail & Related papers  (2023-06-01T02:25:55Z)
• Objectives Matter: Understanding the Impact of Self-Supervised Objectives on Vision Transformer Representations [13.437097059358067]
  We show that reconstruction-based learning features are significantly dissimilar to joint-embedding based learning features. We find that joint-embedding features yield better linear probe transfer for classification because the different objectives drive different distributions of information.
  arXiv  Detail & Related papers  (2023-04-25T18:48:23Z)
• Robust and Controllable Object-Centric Learning through Energy-based Models [95.68748828339059]
  ours is a conceptually simple and general approach to learning object-centric representations through an energy-based model. We show that ours can be easily integrated into existing architectures and can effectively extract high-quality object-centric representations.
  arXiv  Detail & Related papers  (2022-10-11T15:11:15Z)
• Towards Self-Supervised Category-Level Object Pose and Size Estimation [121.28537953301951]
  This work presents a self-supervised framework for category-level object pose and size estimation from a single depth image. We leverage the geometric consistency residing in point clouds of the same shape for self-supervision.
  arXiv  Detail & Related papers  (2022-03-06T06:02:30Z)
• Appearance Consensus Driven Self-Supervised Human Mesh Recovery [67.20942777949793]
  We present a self-supervised human mesh recovery framework to infer human pose and shape from monocular images. We achieve state-of-the-art results on the standard model-based 3D pose estimation benchmarks. The resulting colored mesh prediction opens up the usage of our framework for a variety of appearance-related tasks beyond the pose and shape estimation.
  arXiv  Detail & Related papers  (2020-08-04T05:40:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.