Heterogeneous Image GNN: Graph-Conditioned Diffusion for Image Synthesis

• URL: http://arxiv.org/abs/2502.01309v1
• Date: Mon, 03 Feb 2025 12:36:14 GMT
• Title: Heterogeneous Image GNN: Graph-Conditioned Diffusion for Image Synthesis
• Authors: Rupert Menneer, Christos Margadji, Sebastian W. Pattinson,
• Abstract summary: This paper presents Heterogeneous Image Graphs (HIG), a novel representation that models conditioning variables and target images as two interconnected graphs. We also propose a magnitude-preserving GNN that integrates the HIG into the existing EDM2 diffusion model using a ControlNet approach.
• Score: 0.0
• License:
• Abstract: We introduce a novel method for conditioning diffusion-based image synthesis models with heterogeneous graph data. Existing approaches typically incorporate conditioning variables directly into model architectures, either through cross-attention layers that attend to text latents or image concatenation that spatially restrict generation. However, these methods struggle to handle complex scenarios involving diverse, relational conditioning variables, which are more naturally represented as unstructured graphs. This paper presents Heterogeneous Image Graphs (HIG), a novel representation that models conditioning variables and target images as two interconnected graphs, enabling efficient handling of variable-length conditioning inputs and their relationships. We also propose a magnitude-preserving GNN that integrates the HIG into the existing EDM2 diffusion model using a ControlNet approach. Our approach improves upon the SOTA on a variety of conditioning inputs for the COCO-stuff and Visual Genome datasets, and showcases the ability to condition on graph attributes and relationships represented by edges in the HIG.

Related papers

• DiffGraph: Heterogeneous Graph Diffusion Model [16.65576765238224]
  Graph Neural Networks (GNNs) have revolutionized graph-structured data modeling, yet traditional GNNs struggle with complex heterogeneous structures prevalent in real-world scenarios. We present the Heterogeneous Graph Diffusion Model (DiffGraph), a pioneering framework that introduces an innovative cross-view denoising strategy. At its core, DiffGraph features a sophisticated latent heterogeneous graph diffusion mechanism, implementing a novel forward and backward diffusion process for superior noise management.
  arXiv  Detail & Related papers  (2025-01-04T15:30:48Z)
• Training-free Heterogeneous Graph Condensation via Data Selection [74.06562124781104]
  We present the first Training underlineFree Heterogeneous Graph Condensation method, termed FreeHGC, facilitating both efficient and high-quality generation of heterogeneous condensed graphs. Specifically, we reformulate the heterogeneous graph condensation problem as a data selection issue, offering a new perspective for assessing and condensing representative nodes and edges in the heterogeneous graphs.
  arXiv  Detail & Related papers  (2024-12-20T02:49:32Z)
• Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesis [43.481539150288434]
  This work introduces a new family of. factor graph Diffusion Models (FG-DMs) FG-DMs models the joint distribution of. images and conditioning variables, such as semantic, sketch,. deep or normal maps via a factor graph decomposition.
  arXiv  Detail & Related papers  (2024-10-29T00:54:00Z)
• SeaDAG: Semi-autoregressive Diffusion for Conditional Directed Acyclic Graph Generation [83.52157311471693]
  We introduce SeaDAG, a semi-autoregressive diffusion model for conditional generation of Directed Acyclic Graphs (DAGs) Unlike conventional autoregressive generation that lacks a global graph structure view, our method maintains a complete graph structure at each diffusion step. We explicitly train the model to learn graph conditioning with a condition loss, which enhances the diffusion model's capacity to generate realistic DAGs.
  arXiv  Detail & Related papers  (2024-10-21T15:47:03Z)
• Graph Transformer GANs with Graph Masked Modeling for Architectural Layout Generation [153.92387500677023]
  We present a novel graph Transformer generative adversarial network (GTGAN) to learn effective graph node relations. The proposed graph Transformer encoder combines graph convolutions and self-attentions in a Transformer to model both local and global interactions. We also propose a novel self-guided pre-training method for graph representation learning.
  arXiv  Detail & Related papers  (2024-01-15T14:36:38Z)
• Joint Generative Modeling of Scene Graphs and Images via Diffusion Models [37.788957749123725]
  We present a novel generative task: joint scene graph - image generation. We introduce a novel diffusion model, DiffuseSG, that jointly models the adjacency matrix along with heterogeneous node and edge attributes. With a graph transformer being the denoiser, DiffuseSG successively denoises the scene graph representation in a continuous space and discretizes the final representation to generate the clean scene graph.
  arXiv  Detail & Related papers  (2024-01-02T10:10:29Z)
• Spatial-spectral Hyperspectral Image Classification via Multiple Random Anchor Graphs Ensemble Learning [88.60285937702304]
  This paper proposes a novel spatial-spectral HSI classification method via multiple random anchor graphs ensemble learning (RAGE) Firstly, the local binary pattern is adopted to extract the more descriptive features on each selected band, which preserves local structures and subtle changes of a region. Secondly, the adaptive neighbors assignment is introduced in the construction of anchor graph, to reduce the computational complexity.
  arXiv  Detail & Related papers  (2021-03-25T09:31:41Z)
• Lightweight, Dynamic Graph Convolutional Networks for AMR-to-Text Generation [56.73834525802723]
  Lightweight Dynamic Graph Convolutional Networks (LDGCNs) are proposed. LDGCNs capture richer non-local interactions by synthesizing higher order information from the input graphs. We develop two novel parameter saving strategies based on the group graph convolutions and weight tied convolutions to reduce memory usage and model complexity.
  arXiv  Detail & Related papers  (2020-10-09T06:03:46Z)
• Heterogeneous Graph Transformer [49.675064816860505]
  Heterogeneous Graph Transformer (HGT) architecture for modeling Web-scale heterogeneous graphs. To handle dynamic heterogeneous graphs, we introduce the relative temporal encoding technique into HGT. To handle Web-scale graph data, we design the heterogeneous mini-batch graph sampling algorithm---HGSampling---for efficient and scalable training.
  arXiv  Detail & Related papers  (2020-03-03T04:49:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.