Generating metamers of human scene understanding

• URL: http://arxiv.org/abs/2601.11675v1
• Date: Fri, 16 Jan 2026 06:24:59 GMT
• Title: Generating metamers of human scene understanding
• Authors: Ritik Raina, Abe Leite, Alexandros Graikos, Seoyoung Ahn, Dimitris Samaras, Gregory J. Zelinsky,
• Abstract summary: We introduce MetamerGen, a tool for generating scenes that are aligned with latent human scene representations.<n> generating images from both high and low resolution (i.e. "foveated") inputs constitutes a novel image-to-image synthesis problem.<n>We find that high-level semantic alignment most strongly predicts metamerism when the generated scenes are conditioned on viewers' own fixated regions.
• Score: 67.68406304999473
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human vision combines low-resolution "gist" information from the visual periphery with sparse but high-resolution information from fixated locations to construct a coherent understanding of a visual scene. In this paper, we introduce MetamerGen, a tool for generating scenes that are aligned with latent human scene representations. MetamerGen is a latent diffusion model that combines peripherally obtained scene gist information with information obtained from scene-viewing fixations to generate image metamers for what humans understand after viewing a scene. Generating images from both high and low resolution (i.e. "foveated") inputs constitutes a novel image-to-image synthesis problem, which we tackle by introducing a dual-stream representation of the foveated scenes consisting of DINOv2 tokens that fuse detailed features from fixated areas with peripherally degraded features capturing scene context. To evaluate the perceptual alignment of MetamerGen generated images to latent human scene representations, we conducted a same-different behavioral experiment where participants were asked for a "same" or "different" response between the generated and the original image. With that, we identify scene generations that are indeed metamers for the latent scene representations formed by the viewers. MetamerGen is a powerful tool for understanding scene understanding. Our proof-of-concept analyses uncovered specific features at multiple levels of visual processing that contributed to human judgments. While it can generate metamers even conditioned on random fixations, we find that high-level semantic alignment most strongly predicts metamerism when the generated scenes are conditioned on viewers' own fixated regions.

Related papers

• From Parts to Whole: A Unified Reference Framework for Controllable Human Image Generation [19.096741614175524]
  Parts2Whole is a novel framework designed for generating customized portraits from multiple reference images. We first develop a semantic-aware appearance encoder to retain details of different human parts. Second, our framework supports multi-image conditioned generation through a shared self-attention mechanism.
  arXiv  Detail & Related papers  (2024-04-23T17:56:08Z)
• CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion [83.30168660888913]
  We present CommonScenes, a fully generative model that converts scene graphs into corresponding controllable 3D scenes. Our pipeline consists of two branches, one predicting the overall scene layout via a variational auto-encoder and the other generating compatible shapes. The generated scenes can be manipulated by editing the input scene graph and sampling the noise in the diffusion model.
  arXiv  Detail & Related papers  (2023-05-25T17:39:13Z)
• Identity-Preserving Talking Face Generation with Landmark and Appearance Priors [106.79923577700345]
  Existing person-generic methods have difficulty in generating realistic and lip-synced videos. We propose a two-stage framework consisting of audio-to-landmark generation and landmark-to-video rendering procedures. Our method can produce more realistic, lip-synced, and identity-preserving videos than existing person-generic talking face generation methods.
  arXiv  Detail & Related papers  (2023-05-15T01:31:32Z)
• Contextually-rich human affect perception using multimodal scene information [36.042369831043686]
  We leverage pretrained vision-language (VLN) models to extract descriptions of foreground context from images. We propose a multimodal context fusion (MCF) module to combine foreground cues with the visual scene and person-based contextual information for emotion prediction. We show the effectiveness of our proposed modular design on two datasets associated with natural scenes and TV shows.
  arXiv  Detail & Related papers  (2023-03-13T07:46:41Z)
• Semantically Consistent Person Image Generation [18.73832646369506]
  We propose a data-driven approach for context-aware person image generation.<n>In our method, the position, scale, and appearance of the generated person are semantically conditioned on the existing persons in the scene.
  arXiv  Detail & Related papers  (2023-02-28T16:34:55Z)
• Understanding Cross-modal Interactions in V&L Models that Generate Scene Descriptions [3.7957452405531256]
  This paper explores the potential of a state-of-the-art Vision and Language model, VinVL, to caption images at the scene level. We show (3) that a small amount of curated data suffices to generate scene descriptions without losing the capability to identify object-level concepts in the scene. We discuss the parallels between these results and insights from computational and cognitive science research on scene perception.
  arXiv  Detail & Related papers  (2022-11-09T15:33:51Z)
• Advances in Neural Rendering [115.05042097988768]
  This report focuses on methods that combine classical rendering with learned 3D scene representations. A key advantage of these methods is that they are 3D-consistent by design, enabling applications such as novel viewpoint of a captured scene. In addition to methods that handle static scenes, we cover neural scene representations for modeling non-rigidly deforming objects.
  arXiv  Detail & Related papers  (2021-11-10T18:57:01Z)
• Neural Scene Graphs for Dynamic Scenes [57.65413768984925]
  We present the first neural rendering method that decomposes dynamic scenes into scene graphs. We learn implicitly encoded scenes, combined with a jointly learned latent representation to describe objects with a single implicit function.
  arXiv  Detail & Related papers  (2020-11-20T12:37:10Z)
• Sketching Image Gist: Human-Mimetic Hierarchical Scene Graph Generation [98.34909905511061]
  We argue that a desirable scene graph should be hierarchically constructed, and introduce a new scheme for modeling scene graph. To generate a scene graph based on HET, we parse HET with a Hybrid Long Short-Term Memory (Hybrid-LSTM) which specifically encodes hierarchy and siblings context. To further prioritize key relations in the scene graph, we devise a Relation Ranking Module (RRM) to dynamically adjust their rankings.
  arXiv  Detail & Related papers  (2020-07-17T05:12:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.