Human Gaze Boosts Object-Centered Representation Learning

• URL: http://arxiv.org/abs/2501.02966v1
• Date: Mon, 06 Jan 2025 12:21:40 GMT
• Title: Human Gaze Boosts Object-Centered Representation Learning
• Authors: Timothy Schaumlöffel, Arthur Aubret, Gemma Roig, Jochen Triesch,
• Abstract summary: Recent self-supervised learning models trained on human-like egocentric visual inputs substantially underperform on image recognition tasks compared to humans. Here, we investigate whether focusing on central visual information boosts egocentric visual object learning. Our experiments demonstrate that focusing on central vision leads to better object-centered representations.
• Score: 7.473473243713322
• License:
• Abstract: Recent self-supervised learning (SSL) models trained on human-like egocentric visual inputs substantially underperform on image recognition tasks compared to humans. These models train on raw, uniform visual inputs collected from head-mounted cameras. This is different from humans, as the anatomical structure of the retina and visual cortex relatively amplifies the central visual information, i.e. around humans' gaze location. This selective amplification in humans likely aids in forming object-centered visual representations. Here, we investigate whether focusing on central visual information boosts egocentric visual object learning. We simulate 5-months of egocentric visual experience using the large-scale Ego4D dataset and generate gaze locations with a human gaze prediction model. To account for the importance of central vision in humans, we crop the visual area around the gaze location. Finally, we train a time-based SSL model on these modified inputs. Our experiments demonstrate that focusing on central vision leads to better object-centered representations. Our analysis shows that the SSL model leverages the temporal dynamics of the gaze movements to build stronger visual representations. Overall, our work marks a significant step toward bio-inspired learning of visual representations.

Related papers

• When Does Perceptual Alignment Benefit Vision Representations? [76.32336818860965]
  We investigate how aligning vision model representations to human perceptual judgments impacts their usability. We find that aligning models to perceptual judgments yields representations that improve upon the original backbones across many downstream tasks. Our results suggest that injecting an inductive bias about human perceptual knowledge into vision models can contribute to better representations.
  arXiv  Detail & Related papers  (2024-10-14T17:59:58Z)
• Visual attention information can be traced on cortical response but not on the retina: evidence from electrophysiological mouse data using natural images as stimuli [0.0]
  In primary visual cortex (V1), a subset of around $10%$ of the neurons responds differently to salient versus non-salient visual regions. It appears that the retina remains naive concerning visual attention; cortical response gets to interpret visual attention information.
  arXiv  Detail & Related papers  (2023-08-01T13:09:48Z)
• BI AVAN: Brain inspired Adversarial Visual Attention Network [67.05560966998559]
  We propose a brain-inspired adversarial visual attention network (BI-AVAN) to characterize human visual attention directly from functional brain activity. Our model imitates the biased competition process between attention-related/neglected objects to identify and locate the visual objects in a movie frame the human brain focuses on in an unsupervised manner.
  arXiv  Detail & Related papers  (2022-10-27T22:20:36Z)
• Guiding Visual Attention in Deep Convolutional Neural Networks Based on Human Eye Movements [0.0]
  Deep Convolutional Neural Networks (DCNNs) were originally inspired by principles of biological vision. Recent advances in deep learning seem to decrease this similarity. We investigate a purely data-driven approach to obtain useful models.
  arXiv  Detail & Related papers  (2022-06-21T17:59:23Z)
• Human Eyes Inspired Recurrent Neural Networks are More Robust Against Adversarial Noises [7.689542442882423]
  We designed a dual-stream vision model inspired by the human brain. This model features retina-like input layers and includes two streams: one determining the next point of focus (the fixation), while the other interprets the visuals surrounding the fixation. We evaluated this model against various benchmarks in terms of object recognition, gaze behavior and adversarial robustness.
  arXiv  Detail & Related papers  (2022-06-15T03:44:42Z)
• Peripheral Vision Transformer [52.55309200601883]
  We take a biologically inspired approach and explore to model peripheral vision in deep neural networks for visual recognition. We propose to incorporate peripheral position encoding to the multi-head self-attention layers to let the network learn to partition the visual field into diverse peripheral regions given training data. We evaluate the proposed network, dubbed PerViT, on the large-scale ImageNet dataset and systematically investigate the inner workings of the model for machine perception.
  arXiv  Detail & Related papers  (2022-06-14T12:47:47Z)
• Embodied vision for learning object representations [4.211128681972148]
  We show that visual statistics mimicking those of a toddler improve object recognition accuracy in both familiar and novel environments. We argue that this effect is caused by the reduction of features extracted in the background, a neural network bias for large features in the image and a greater similarity between novel and familiar background regions.
  arXiv  Detail & Related papers  (2022-05-12T16:36:27Z)
• Gaze Perception in Humans and CNN-Based Model [66.89451296340809]
  We compare how a CNN (convolutional neural network) based model of gaze and humans infer the locus of attention in images of real-world scenes. We show that compared to the model, humans' estimates of the locus of attention are more influenced by the context of the scene.
  arXiv  Detail & Related papers  (2021-04-17T04:52:46Z)
• What Can You Learn from Your Muscles? Learning Visual Representation from Human Interactions [50.435861435121915]
  We use human interaction and attention cues to investigate whether we can learn better representations compared to visual-only representations. Our experiments show that our "muscly-supervised" representation outperforms a visual-only state-of-the-art method MoCo.
  arXiv  Detail & Related papers  (2020-10-16T17:46:53Z)
• VisualEchoes: Spatial Image Representation Learning through Echolocation [97.23789910400387]
  Several animal species (e.g., bats, dolphins, and whales) and even visually impaired humans have the remarkable ability to perform echolocation. We propose a novel interaction-based representation learning framework that learns useful visual features via echolocation. Our work opens a new path for representation learning for embodied agents, where supervision comes from interacting with the physical world.
  arXiv  Detail & Related papers  (2020-05-04T16:16:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.