Modeling Visual Memorability Assessment with Autoencoders Reveals Characteristics of Memorable Images

• URL: http://arxiv.org/abs/2410.15235v1
• Date: Sat, 19 Oct 2024 22:58:33 GMT
• Title: Modeling Visual Memorability Assessment with Autoencoders Reveals Characteristics of Memorable Images
• Authors: Elham Bagheri, Yalda Mohsenzadeh,
• Abstract summary: Image memorability refers to the phenomenon where certain images are more likely to be remembered than others. We modeled the subjective experience of visual memorability using an autoencoder based on VGG16 Convolutional Neural Networks (CNNs) We investigated the relationship between memorability and reconstruction error, assessed latent space representations distinctiveness, and developed a Gated Recurrent Unit (GRU) model to predict memorability likelihood.
• Score: 2.4861619769660637
• License:
• Abstract: Background: Image memorability refers to the phenomenon where certain images are more likely to be remembered than others. It is a quantifiable and intrinsic image attribute, defined as the likelihood of being remembered upon a single exposure. Despite advances in understanding human visual perception and memory, it is unclear what features contribute to an image's memorability. To address this question, we propose a deep learning-based computational modeling approach. Methods: We modeled the subjective experience of visual memorability using an autoencoder based on VGG16 Convolutional Neural Networks (CNNs). The model was trained on images for one epoch, to simulate the single-exposure condition used in human memory tests. We investigated the relationship between memorability and reconstruction error, assessed latent space representations distinctiveness, and developed a Gated Recurrent Unit (GRU) model to predict memorability likelihood. Interpretability analysis was conducted to identify key image characteristics contributing to memorability. Results: Our results demonstrate a significant correlation between the images memorability score and autoencoder's reconstruction error, and the robust predictive performance of its latent representations. Distinctiveness in these representations correlated significantly with memorability. Additionally, certain visual characteristics, such as strong contrasts, distinctive objects, and prominent foreground elements were among the features contributing to image memorability in our model. Conclusions: Images with unique features that challenge the autoencoder's capacity are inherently more memorable. Moreover, these memorable images are distinct from others the model has encountered, and the latent space of the encoder contains features predictive of memorability.

Related papers

• When Does Perceptual Alignment Benefit Vision Representations? [76.32336818860965]
  We investigate how aligning vision model representations to human perceptual judgments impacts their usability. We find that aligning models to perceptual judgments yields representations that improve upon the original backbones across many downstream tasks. Our results suggest that injecting an inductive bias about human perceptual knowledge into vision models can contribute to better representations.
  arXiv  Detail & Related papers  (2024-10-14T17:59:58Z)
• ArtVLM: Attribute Recognition Through Vision-Based Prefix Language Modeling [32.55352435358949]
  We propose a sentence generation-based retrieval formulation for attribute recognition. For each attribute to be recognized on an image, we measure the visual-conditioned probability of generating a short sentence. We demonstrate through experiments that generative retrieval consistently outperforms contrastive retrieval on two visual reasoning datasets.
  arXiv  Detail & Related papers  (2024-08-07T21:44:29Z)
• Counterfactual Image Editing [54.21104691749547]
  Counterfactual image editing is an important task in generative AI, which asks how an image would look if certain features were different. We formalize the counterfactual image editing task using formal language, modeling the causal relationships between latent generative factors and images. We develop an efficient algorithm to generate counterfactual images by leveraging neural causal models.
  arXiv  Detail & Related papers  (2024-02-07T20:55:39Z)
• Anomaly Score: Evaluating Generative Models and Individual Generated Images based on Complexity and Vulnerability [21.355484227864466]
  We investigate the relationship between the representation space and input space around generated images. We introduce a new metric to evaluating image-generative models called anomaly score (AS)
  arXiv  Detail & Related papers  (2023-12-17T07:33:06Z)
• Attribute-Aware Deep Hashing with Self-Consistency for Large-Scale Fine-Grained Image Retrieval [65.43522019468976]
  We propose attribute-aware hashing networks with self-consistency for generating attribute-aware hash codes. We develop an encoder-decoder structure network of a reconstruction task to unsupervisedly distill high-level attribute-specific vectors. Our models are equipped with a feature decorrelation constraint upon these attribute vectors to strengthen their representative abilities.
  arXiv  Detail & Related papers  (2023-11-21T08:20:38Z)
• Cross-Image Attention for Zero-Shot Appearance Transfer [68.43651329067393]
  We introduce a cross-image attention mechanism that implicitly establishes semantic correspondences across images. We harness three mechanisms that either manipulate the noisy latent codes or the model's internal representations throughout the denoising process. Experiments show that our method is effective across a wide range of object categories and is robust to variations in shape, size, and viewpoint.
  arXiv  Detail & Related papers  (2023-11-06T18:33:24Z)
• Reconstruction-guided attention improves the robustness and shape processing of neural networks [5.156484100374057]
  We build an iterative encoder-decoder network that generates an object reconstruction and uses it as top-down attentional feedback. Our model shows strong generalization performance against various image perturbations. Our study shows that modeling reconstruction-based feedback endows AI systems with a powerful attention mechanism.
  arXiv  Detail & Related papers  (2022-09-27T18:32:22Z)
• A domain adaptive deep learning solution for scanpath prediction of paintings [66.46953851227454]
  This paper focuses on the eye-movement analysis of viewers during the visual experience of a certain number of paintings. We introduce a new approach to predicting human visual attention, which impacts several cognitive functions for humans. The proposed new architecture ingests images and returns scanpaths, a sequence of points featuring a high likelihood of catching viewers' attention.
  arXiv  Detail & Related papers  (2022-09-22T22:27:08Z)
• Exploring CLIP for Assessing the Look and Feel of Images [87.97623543523858]
  We introduce Contrastive Language-Image Pre-training (CLIP) models for assessing both the quality perception (look) and abstract perception (feel) of images in a zero-shot manner. Our results show that CLIP captures meaningful priors that generalize well to different perceptual assessments.
  arXiv  Detail & Related papers  (2022-07-25T17:58:16Z)
• Associative Memories via Predictive Coding [37.59398215921529]
  Associative memories in the brain receive and store patterns of activity registered by the sensory neurons. We present a novel neural model for realizing associative memories based on a hierarchical generative network that receives external stimuli via sensory neurons.
  arXiv  Detail & Related papers  (2021-09-16T15:46:26Z)
• Generating Memorable Images Based on Human Visual Memory Schemas [9.986390874391095]
  This research study proposes using Generative Adversarial Networks (GAN) to generate memorable or non-memorable images of scenes. The memorability of the generated images is evaluated by modelling Visual Memorys (VMS), which correspond to mental representations that human observers use to encode an image into memory.
  arXiv  Detail & Related papers  (2020-05-06T17:23:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.