Evaluating Multiview Object Consistency in Humans and Image Models

• URL: http://arxiv.org/abs/2409.05862v2
• Date: Tue, 10 Sep 2024 02:28:40 GMT
• Title: Evaluating Multiview Object Consistency in Humans and Image Models
• Authors: Tyler Bonnen, Stephanie Fu, Yutong Bai, Thomas O'Connell, Yoni Friedman, Nancy Kanwisher, Joshua B. Tenenbaum, Alexei A. Efros,
• Abstract summary: We leverage an experimental design from the cognitive sciences which requires zero-shot visual inferences about object shape. We collect 35K trials of behavioral data from over 500 participants. We then evaluate the performance of common vision models.
• Score: 68.36073530804296
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We introduce a benchmark to directly evaluate the alignment between human observers and vision models on a 3D shape inference task. We leverage an experimental design from the cognitive sciences which requires zero-shot visual inferences about object shape: given a set of images, participants identify which contain the same/different objects, despite considerable viewpoint variation. We draw from a diverse range of images that include common objects (e.g., chairs) as well as abstract shapes (i.e., procedurally generated `nonsense' objects). After constructing over 2000 unique image sets, we administer these tasks to human participants, collecting 35K trials of behavioral data from over 500 participants. This includes explicit choice behaviors as well as intermediate measures, such as reaction time and gaze data. We then evaluate the performance of common vision models (e.g., DINOv2, MAE, CLIP). We find that humans outperform all models by a wide margin. Using a multi-scale evaluation approach, we identify underlying similarities and differences between models and humans: while human-model performance is correlated, humans allocate more time/processing on challenging trials. All images, data, and code can be accessed via our project page.

Related papers

• When Does Perceptual Alignment Benefit Vision Representations? [76.32336818860965]
  We investigate how aligning vision model representations to human perceptual judgments impacts their usability. We find that aligning models to perceptual judgments yields representations that improve upon the original backbones across many downstream tasks. Our results suggest that injecting an inductive bias about human perceptual knowledge into vision models can contribute to better representations.
  arXiv  Detail & Related papers  (2024-10-14T17:59:58Z)
• UniHuman: A Unified Model for Editing Human Images in the Wild [49.896715833075106]
  We propose UniHuman, a unified model that addresses multiple facets of human image editing in real-world settings. To enhance the model's generation quality and generalization capacity, we leverage guidance from human visual encoders. In user studies, UniHuman is preferred by the users in an average of 77% of cases.
  arXiv  Detail & Related papers  (2023-12-22T05:00:30Z)
• UniAR: A Unified model for predicting human Attention and Responses on visual content [12.281060227170792]
  We propose UniAR -- a unified model of human attention and preference behavior across diverse visual content. We train UniAR on diverse public datasets spanning natural images, webpages, and graphic designs, and achieve SOTA performance on multiple benchmarks. Potential applications include providing instant feedback on the effectiveness of UIs/visual content, and enabling designers and content-creation models to optimize their creation for human-centric improvements.
  arXiv  Detail & Related papers  (2023-12-15T19:57:07Z)
• Heuristic Vision Pre-Training with Self-Supervised and Supervised Multi-Task Learning [0.0]
  We propose a novel pre-training framework by adopting both self-supervised and supervised visual pre-text tasks in a multi-task manner. Results show that our pre-trained models can deliver results on par with or better than state-of-the-art (SOTA) results on multiple visual tasks.
  arXiv  Detail & Related papers  (2023-10-11T14:06:04Z)
• Full-Body Articulated Human-Object Interaction [61.01135739641217]
  CHAIRS is a large-scale motion-captured f-AHOI dataset consisting of 16.2 hours of versatile interactions. CHAIRS provides 3D meshes of both humans and articulated objects during the entire interactive process. By learning the geometrical relationships in HOI, we devise the very first model that leverage human pose estimation.
  arXiv  Detail & Related papers  (2022-12-20T19:50:54Z)
• Reconstructing Action-Conditioned Human-Object Interactions Using Commonsense Knowledge Priors [42.17542596399014]
  We present a method for inferring diverse 3D models of human-object interactions from images. Our method extracts high-level commonsense knowledge from large language models. We quantitatively evaluate the inferred 3D models on a large human-object interaction dataset.
  arXiv  Detail & Related papers  (2022-09-06T13:32:55Z)
• Neural Novel Actor: Learning a Generalized Animatable Neural Representation for Human Actors [98.24047528960406]
  We propose a new method for learning a generalized animatable neural representation from a sparse set of multi-view imagery of multiple persons. The learned representation can be used to synthesize novel view images of an arbitrary person from a sparse set of cameras, and further animate them with the user's pose control.
  arXiv  Detail & Related papers  (2022-08-25T07:36:46Z)
• Benchmarking human visual search computational models in natural scenes: models comparison and reference datasets [0.0]
  We select publicly available state-of-the-art visual search models in natural scenes and evaluate them on different datasets. We propose an improvement to the Ideal Bayesian Searcher through a combination with a neural network-based visual search model.
  arXiv  Detail & Related papers  (2021-12-10T19:56:45Z)
• Learning Online Visual Invariances for Novel Objects via Supervised and Self-Supervised Training [0.76146285961466]
  This paper assesses whether standard CNNs can support human-like online invariance by training models to recognize images of synthetic 3D objects that undergo several transformations. We show that standard supervised CNNs trained on transformed objects can acquire strong invariances on novel classes even when trained with as few as 50 objects taken from 10 classes.
  arXiv  Detail & Related papers  (2021-10-04T14:29:43Z)
• What Can You Learn from Your Muscles? Learning Visual Representation from Human Interactions [50.435861435121915]
  We use human interaction and attention cues to investigate whether we can learn better representations compared to visual-only representations. Our experiments show that our "muscly-supervised" representation outperforms a visual-only state-of-the-art method MoCo.
  arXiv  Detail & Related papers  (2020-10-16T17:46:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.