Evaluating generative networks using Gaussian mixtures of image features

• URL: http://arxiv.org/abs/2110.05240v1
• Date: Fri, 8 Oct 2021 17:58:29 GMT
• Title: Evaluating generative networks using Gaussian mixtures of image features
• Abstract summary: We develop a measure for evaluating the performance of generative networks given two sets of images. We show the advantages of WaM over FID, including how FID is more sensitive than WaM to image perturbations.
• Score: 28.78445970020791
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We develop a measure for evaluating the performance of generative networks given two sets of images. A popular performance measure currently used to do this is the Fr\'echet Inception Distance (FID). However, FID assumes that images featurized using the penultimate layer of Inception-v3 follow a Gaussian distribution. This assumption allows FID to be easily computed, since FID uses the 2-Wasserstein distance of two Gaussian distributions fitted to the featurized images. However, we show that Inception-v3 features of the ImageNet dataset are not Gaussian; in particular, each marginal is not Gaussian. To remedy this problem, we model the featurized images using Gaussian mixture models (GMMs) and compute the 2-Wasserstein distance restricted to GMMs. We define a performance measure, which we call WaM, on two sets of images by using Inception-v3 (or another classifier) to featurize the images, estimate two GMMs, and use the restricted 2-Wasserstein distance to compare the GMMs. We experimentally show the advantages of WaM over FID, including how FID is more sensitive than WaM to image perturbations. By modelling the non-Gaussian features obtained from Inception-v3 as GMMs and using a GMM metric, we can more accurately evaluate generative network performance.

Related papers

• Disambiguating 2D-3D Correspondences in Gaussian Splatting-based Feature Fields for Visual Localization [36.99946317601775]
  This paper highlights that photometrically optimized Feature Fields (GSFFs) are inherently ill-suited for 2D-3D matching.<n>To address these issues, we propose SplitGS-Loc, a specialized localization framework that disambiates 2D-3D correspondences by exploiting Gaussian attributes.
  arXiv  Detail & Related papers  (2026-05-08T07:01:30Z)
• ShelfGaussian: Shelf-Supervised Open-Vocabulary Gaussian-based 3D Scene Understanding [7.610505486431266]
  We introduce ShelfGaussian, an open-vocabulary multi-modal Gaussian-based 3D scene understanding framework supervised by off-the-shelf vision foundation models.<n>Existing methods either model objects as closed-set semantic Gaussians supervised by annotated 3D labels, neglecting their rendering ability, or learn open-set Gaussian representations via purely 2D self-supervision.
  arXiv  Detail & Related papers  (2025-12-03T02:06:09Z)
• iGaussian: Real-Time Camera Pose Estimation via Feed-Forward 3D Gaussian Splatting Inversion [62.09575122593993]
  iGaussian is a two-stage feed-forward framework that achieves real-time camera pose estimation through direct 3D Gaussian inversion.<n> Experimental results on the NeRF Synthetic, Mip-NeRF 360, and T&T+DB datasets demonstrate a significant performance improvement over previous methods.
  arXiv  Detail & Related papers  (2025-11-18T05:22:22Z)
• SHeaP: Self-Supervised Head Geometry Predictor Learned via 2D Gaussians [55.813327441814344]
  3D reconstruction of human heads from monocular images and videos underlies numerous visual applications. Previous methods have sought to learn from abundant 2D videos in a self-supervised manner. We propose SHeaP (Self-supervised Head Geometry Predictor Learned via 2D Gaussians)
  arXiv  Detail & Related papers  (2025-04-16T17:55:02Z)
• GaussianToken: An Effective Image Tokenizer with 2D Gaussian Splatting [64.84383010238908]
  We propose an effective image tokenizer with 2D Gaussian Splatting as a solution. In general, our framework integrates the local influence of 2D Gaussian distribution into the discrete space. Competitive reconstruction performances on CIFAR, Mini-Net, and ImageNet-1K demonstrate the effectiveness of our framework.
  arXiv  Detail & Related papers  (2025-01-26T17:56:11Z)
• NovelGS: Consistent Novel-view Denoising via Large Gaussian Reconstruction Model [57.92709692193132]
  NovelGS is a diffusion model for Gaussian Splatting given sparse-view images. We leverage the novel view denoising through a transformer-based network to generate 3D Gaussians.
  arXiv  Detail & Related papers  (2024-11-25T07:57:17Z)
• Normalizing Flow-Based Metric for Image Generation [4.093503153499691]
  We propose two new evaluation metrics to assess realness of generated images based on normalizing flows. Because normalizing flows can be used to compute the exact likelihood, the proposed metrics assess how closely generated images align with the distribution of real images from a given domain.
  arXiv  Detail & Related papers  (2024-10-02T20:09:58Z)
• Image-GS: Content-Adaptive Image Representation via 2D Gaussians [55.15950594752051]
  We propose Image-GS, a content-adaptive image representation. Using anisotropic 2D Gaussians as the basis, Image-GS shows high memory efficiency, supports fast random access, and offers a natural level of detail stack. General efficiency and fidelity of Image-GS are validated against several recent neural image representations and industry-standard texture compressors. We hope this research offers insights for developing new applications that require adaptive quality and resource control, such as machine perception, asset streaming, and content generation.
  arXiv  Detail & Related papers  (2024-07-02T00:45:21Z)
• Deep Gaussian mixture model for unsupervised image segmentation [1.3654846342364308]
  In many tasks sufficient pixel-level labels are very difficult to obtain. We propose a method which combines a Gaussian mixture model (GMM) with unsupervised deep learning techniques. We demonstrate the advantages of our method in various experiments on the example of infarct segmentation on multi-sequence MRI images.
  arXiv  Detail & Related papers  (2024-04-18T15:20:59Z)
• Cramer Type Distances for Learning Gaussian Mixture Models by Gradient Descent [0.0]
  As of today, few known algorithms can fit or learn Gaussian mixture models. We propose a distance function called Sliced Cram'er 2-distance for learning general multivariate GMMs. These features are especially useful for distributional reinforcement learning and Deep Q Networks.
  arXiv  Detail & Related papers  (2023-07-13T13:43:02Z)
• GMConv: Modulating Effective Receptive Fields for Convolutional Kernels [52.50351140755224]
  In convolutional neural networks, the convolutions are performed using a square kernel with a fixed N $times$ N receptive field (RF) Inspired by the property that ERFs typically exhibit a Gaussian distribution, we propose a Gaussian Mask convolutional kernel (GMConv) in this work. Our GMConv can directly replace the standard convolutions in existing CNNs and can be easily trained end-to-end by standard back-propagation.
  arXiv  Detail & Related papers  (2023-02-09T10:17:17Z)
• Hypercomplex Image-to-Image Translation [13.483068375377362]
  Image-to-image translation (I2I) aims at transferring the content representation from an input domain to an output one. Recent I2I generative models, which gain outstanding results in this task, comprise a set of diverse deep networks each with tens of million parameters. We propose to leverage hypercomplex algebra properties to define lightweight I2I generative models capable of preserving pre-existing relations among image dimensions.
  arXiv  Detail & Related papers  (2022-05-04T14:28:50Z)
• EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentation [62.210091681352914]
  We study multi-sensor fusion for 3D semantic segmentation for many applications, such as autonomous driving and robotics. In this work, we investigate a collaborative fusion scheme called perception-aware multi-sensor fusion (PMF) We propose a two-stream network to extract features from the two modalities separately. The extracted features are fused by effective residual-based fusion modules.
  arXiv  Detail & Related papers  (2021-06-21T10:47:26Z)
• Image Modeling with Deep Convolutional Gaussian Mixture Models [79.0660895390689]
  We present a new formulation of deep hierarchical Gaussian Mixture Models (GMMs) that is suitable for describing and generating images. DCGMMs avoid this by a stacked architecture of multiple GMM layers, linked by convolution and pooling operations. For generating sharp images with DCGMMs, we introduce a new gradient-based technique for sampling through non-invertible operations like convolution and pooling. Based on the MNIST and FashionMNIST datasets, we validate the DCGMMs model by demonstrating its superiority over flat GMMs for clustering, sampling and outlier detection.
  arXiv  Detail & Related papers  (2021-04-19T12:08:53Z)
• Adaptive Context-Aware Multi-Modal Network for Depth Completion [107.15344488719322]
  We propose to adopt the graph propagation to capture the observed spatial contexts. We then apply the attention mechanism on the propagation, which encourages the network to model the contextual information adaptively. Finally, we introduce the symmetric gated fusion strategy to exploit the extracted multi-modal features effectively. Our model, named Adaptive Context-Aware Multi-Modal Network (ACMNet), achieves the state-of-the-art performance on two benchmarks.
  arXiv  Detail & Related papers  (2020-08-25T06:00:06Z)
• Generate High Resolution Images With Generative Variational Autoencoder [0.0]
  We present a novel neural network to generate high resolution images. We replace the decoder of VAE with a discriminator while using the encoder as it is. We evaluate our network on 3 different datasets: MNIST, LSUN and CelebA dataset.
  arXiv  Detail & Related papers  (2020-08-12T20:15:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.