Network Bending: Expressive Manipulation of Deep Generative Models
- URL: http://arxiv.org/abs/2005.12420v2
- Date: Fri, 12 Mar 2021 15:06:56 GMT
- Title: Network Bending: Expressive Manipulation of Deep Generative Models
- Authors: Terence Broad, Frederic Fol Leymarie, Mick Grierson
- Abstract summary: We introduce a new framework for manipulating and interacting with deep generative models that we call network bending.
We show how it allows for the direct manipulation of semantically meaningful aspects of the generative process as well as allowing for a broad range of expressive outcomes.
- Score: 0.2062593640149624
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: We introduce a new framework for manipulating and interacting with deep
generative models that we call network bending. We present a comprehensive set
of deterministic transformations that can be inserted as distinct layers into
the computational graph of a trained generative neural network and applied
during inference. In addition, we present a novel algorithm for analysing the
deep generative model and clustering features based on their spatial activation
maps. This allows features to be grouped together based on spatial similarity
in an unsupervised fashion. This results in the meaningful manipulation of sets
of features that correspond to the generation of a broad array of semantically
significant features of the generated images. We outline this framework,
demonstrating our results on state-of-the-art deep generative models trained on
several image datasets. We show how it allows for the direct manipulation of
semantically meaningful aspects of the generative process as well as allowing
for a broad range of expressive outcomes.
Related papers
- SpaceMesh: A Continuous Representation for Learning Manifold Surface Meshes [61.110517195874074]
We present a scheme to directly generate manifold, polygonal meshes of complex connectivity as the output of a neural network.
Our key innovation is to define a continuous latent connectivity space at each mesh, which implies the discrete mesh.
In applications, this approach not only yields high-quality outputs from generative models, but also enables directly learning challenging geometry processing tasks such as mesh repair.
arXiv Detail & Related papers (2024-09-30T17:59:03Z) - Towards Scalable and Versatile Weight Space Learning [51.78426981947659]
This paper introduces the SANE approach to weight-space learning.
Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights.
arXiv Detail & Related papers (2024-06-14T13:12:07Z) - Neural Clustering based Visual Representation Learning [61.72646814537163]
Clustering is one of the most classic approaches in machine learning and data analysis.
We propose feature extraction with clustering (FEC), which views feature extraction as a process of selecting representatives from data.
FEC alternates between grouping pixels into individual clusters to abstract representatives and updating the deep features of pixels with current representatives.
arXiv Detail & Related papers (2024-03-26T06:04:50Z) - Learning Structured Output Representations from Attributes using Deep
Conditional Generative Models [0.0]
This paper recreates the Conditional Variational Auto-encoder architecture and trains it on images conditioned on attributes.
We attempt to generate new faces with distinct attributes such as hair color and glasses, as well as different bird species samples.
arXiv Detail & Related papers (2023-04-30T17:25:31Z) - Image Synthesis via Semantic Composition [74.68191130898805]
We present a novel approach to synthesize realistic images based on their semantic layouts.
It hypothesizes that for objects with similar appearance, they share similar representation.
Our method establishes dependencies between regions according to their appearance correlation, yielding both spatially variant and associated representations.
arXiv Detail & Related papers (2021-09-15T02:26:07Z) - Unsupervised Discovery of Disentangled Manifolds in GANs [74.24771216154105]
Interpretable generation process is beneficial to various image editing applications.
We propose a framework to discover interpretable directions in the latent space given arbitrary pre-trained generative adversarial networks.
arXiv Detail & Related papers (2020-11-24T02:18:08Z) - Interpreting Spatially Infinite Generative Models [40.453301580034804]
Recent work has shown that feeding spatial noise vectors into a fully convolutional neural network enables both generation of arbitrary resolution output images and training on arbitrary resolution training images.
We provide a firm theoretical interpretation for infinite spatial generation, by drawing connections to spatial processes.
Experiments on world map generation, panoramic images and texture synthesis verify the ability of $infty$-GAN to efficiently generate images of arbitrary size.
arXiv Detail & Related papers (2020-07-24T09:00:41Z) - Closed-Form Factorization of Latent Semantics in GANs [65.42778970898534]
A rich set of interpretable dimensions has been shown to emerge in the latent space of the Generative Adversarial Networks (GANs) trained for synthesizing images.
In this work, we examine the internal representation learned by GANs to reveal the underlying variation factors in an unsupervised manner.
We propose a closed-form factorization algorithm for latent semantic discovery by directly decomposing the pre-trained weights.
arXiv Detail & Related papers (2020-07-13T18:05:36Z) - Generating Annotated High-Fidelity Images Containing Multiple Coherent
Objects [10.783993190686132]
We propose a multi-object generation framework that can synthesize images with multiple objects without explicitly requiring contextual information.
We demonstrate how coherency and fidelity are preserved with our method through experiments on the Multi-MNIST and CLEVR datasets.
arXiv Detail & Related papers (2020-06-22T11:33:55Z) - Contextual Encoder-Decoder Network for Visual Saliency Prediction [42.047816176307066]
We propose an approach based on a convolutional neural network pre-trained on a large-scale image classification task.
We combine the resulting representations with global scene information for accurately predicting visual saliency.
Compared to state of the art approaches, the network is based on a lightweight image classification backbone.
arXiv Detail & Related papers (2019-02-18T16:15:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.