FunkNN: Neural Interpolation for Functional Generation

• URL: http://arxiv.org/abs/2212.14042v2
• Date: Mon, 3 Apr 2023 12:40:57 GMT
• Title: FunkNN: Neural Interpolation for Functional Generation
• Authors: AmirEhsan Khorashadizadeh, Anadi Chaman, Valentin Debarnot, Ivan Dokmani\'c
• Abstract summary: FunkNN is a new convolutional network which learns to reconstruct continuous images at arbitrary coordinates and can be applied to any image dataset. We show that FunkNN generates high-quality continuous images and exhibits strong out-of-distribution performance thanks to its patch-based design.
• Score: 23.964801524703052
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Can we build continuous generative models which generalize across scales, can be evaluated at any coordinate, admit calculation of exact derivatives, and are conceptually simple? Existing MLP-based architectures generate worse samples than the grid-based generators with favorable convolutional inductive biases. Models that focus on generating images at different scales do better, but employ complex architectures not designed for continuous evaluation of images and derivatives. We take a signal-processing perspective and treat continuous image generation as interpolation from samples. Indeed, correctly sampled discrete images contain all information about the low spatial frequencies. The question is then how to extrapolate the spectrum in a data-driven way while meeting the above design criteria. Our answer is FunkNN -- a new convolutional network which learns how to reconstruct continuous images at arbitrary coordinates and can be applied to any image dataset. Combined with a discrete generative model it becomes a functional generator which can act as a prior in continuous ill-posed inverse problems. We show that FunkNN generates high-quality continuous images and exhibits strong out-of-distribution performance thanks to its patch-based design. We further showcase its performance in several stylized inverse problems with exact spatial derivatives.

Related papers

• Generalized Consistency Trajectory Models for Image Manipulation [59.576781858809355]
  Diffusion models (DMs) excel in unconditional generation, as well as on applications such as image editing and restoration. This work aims to unlock the full potential of consistency trajectory models (CTMs) by proposing generalized CTMs (GCTMs) We discuss the design space of GCTMs and demonstrate their efficacy in various image manipulation tasks such as image-to-image translation, restoration, and editing.
  arXiv  Detail & Related papers  (2024-03-19T07:24:54Z)
• Deep Equilibrium Diffusion Restoration with Parallel Sampling [120.15039525209106]
  Diffusion model-based image restoration (IR) aims to use diffusion models to recover high-quality (HQ) images from degraded images, achieving promising performance. Most existing methods need long serial sampling chains to restore HQ images step-by-step, resulting in expensive sampling time and high computation costs. In this work, we aim to rethink the diffusion model-based IR models through a different perspective, i.e., a deep equilibrium (DEQ) fixed point system, called DeqIR.
  arXiv  Detail & Related papers  (2023-11-20T08:27:56Z)
• Distance Weighted Trans Network for Image Completion [52.318730994423106]
  We propose a new architecture that relies on Distance-based Weighted Transformer (DWT) to better understand the relationships between an image's components. CNNs are used to augment the local texture information of coarse priors. DWT blocks are used to recover certain coarse textures and coherent visual structures.
  arXiv  Detail & Related papers  (2023-10-11T12:46:11Z)
• VTAE: Variational Transformer Autoencoder with Manifolds Learning [144.0546653941249]
  Deep generative models have demonstrated successful applications in learning non-linear data distributions through a number of latent variables. The nonlinearity of the generator implies that the latent space shows an unsatisfactory projection of the data space, which results in poor representation learning. We show that geodesics and accurate computation can substantially improve the performance of deep generative models.
  arXiv  Detail & Related papers  (2023-04-03T13:13:19Z)
• Score-based Diffusion Models in Function Space [140.792362459734]
  Diffusion models have recently emerged as a powerful framework for generative modeling. We introduce a mathematically rigorous framework called Denoising Diffusion Operators (DDOs) for training diffusion models in function space. We show that the corresponding discretized algorithm generates accurate samples at a fixed cost independent of the data resolution.
  arXiv  Detail & Related papers  (2023-02-14T23:50:53Z)
• Using Intermediate Forward Iterates for Intermediate Generator Optimization [14.987013151525368]
  Intermediate Generator Optimization can be incorporated into any standard autoencoder pipeline for the generative task. We show applications of the IGO on two dense predictive tasks viz., image extrapolation, and point cloud denoising.
  arXiv  Detail & Related papers  (2023-02-05T08:46:15Z)
• NeurInt : Learning to Interpolate through Neural ODEs [18.104328632453676]
  We propose a novel generative model that learns a distribution of trajectories between two images. We demonstrate our approach's effectiveness in generating images improved quality as well as its ability to learn a diverse distribution over smooth trajectories for any pair of real source and target images.
  arXiv  Detail & Related papers  (2021-11-07T16:31:18Z)
• Score-based diffusion models for accelerated MRI [35.3148116010546]
  We introduce a way to sample data from a conditional distribution given the measurements, such that the model can be readily used for solving inverse problems in imaging. Our model requires magnitude images only for training, and yet is able to reconstruct complex-valued data, and even extends to parallel imaging.
  arXiv  Detail & Related papers  (2021-10-08T08:42:03Z)
• Deep Neural Networks are Surprisingly Reversible: A Baseline for Zero-Shot Inversion [90.65667807498086]
  This paper presents a zero-shot direct model inversion framework that recovers the input to the trained model given only the internal representation. We empirically show that modern classification models on ImageNet can, surprisingly, be inverted, allowing an approximate recovery of the original 224x224px images from a representation after more than 20 layers.
  arXiv  Detail & Related papers  (2021-07-13T18:01:43Z)
• Image Generators with Conditionally-Independent Pixel Synthesis [10.792933031825527]
  We present a new architecture for image generators, where the color value at each pixel is computed independently. No spatial convolutions or similar operations that propagate information across pixels are involved during the synthesis. We analyze the modeling capabilities of such generators when trained in an adversarial fashion, and observe the new generators to achieve similar generation quality to state-of-the-art convolutional generators.
  arXiv  Detail & Related papers  (2020-11-27T15:16:11Z)
• AE-OT-GAN: Training GANs from data specific latent distribution [21.48007565143911]
  generative adversarial networks (GANs) areprominent models to generate realistic and crisp images. GANs often encounter the mode collapse problems and arehard to train, which comes from approximating the intrinsicdiscontinuous distribution transform map with continuousDNNs. The recently proposed AE-OT model addresses thisproblem by explicitly computing the discontinuous distribu-tion transform map. In this paper, wepropose the AE-OT-GAN model to utilize the advantages ofthe both models: generate high quality images and at the same time overcome the mode collapse/mixture problems.
  arXiv  Detail & Related papers  (2020-01-11T01:18:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.