Neural Capacity Estimators: How Reliable Are They?

• URL: http://arxiv.org/abs/2111.07401v1
• Date: Sun, 14 Nov 2021 18:14:53 GMT
• Title: Neural Capacity Estimators: How Reliable Are They?
• Authors: Farhad Mirkarimi, Stefano Rini, Nariman Farsad
• Abstract summary: We study the performance of mutual information neural estimator (MINE), smoothed mutual information lower-bound estimator (SMILE), and information directed neural estimator (DINE) We evaluate these algorithms in terms of their ability to learn the input distributions that are capacity approaching for the AWGN channel, the optical intensity channel, and peak power-constrained AWGN channel.
• Score: 14.904387585122851
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, several methods have been proposed for estimating the mutual information from sample data using deep neural networks and without the knowing closed form distribution of the data. This class of estimators is referred to as neural mutual information estimators. Although very promising, such techniques have yet to be rigorously bench-marked so as to establish their efficacy, ease of implementation, and stability for capacity estimation which is joint maximization frame-work. In this paper, we compare the different techniques proposed in the literature for estimating capacity and provide a practitioner perspective on their effectiveness. In particular, we study the performance of mutual information neural estimator (MINE), smoothed mutual information lower-bound estimator (SMILE), and directed information neural estimator (DINE) and provide insights on InfoNCE. We evaluated these algorithms in terms of their ability to learn the input distributions that are capacity approaching for the AWGN channel, the optical intensity channel, and peak power-constrained AWGN channel. For both scenarios, we provide insightful comments on various aspects of the training process, such as stability, sensitivity to initialization.

Related papers

• Assessing Neural Network Representations During Training Using Noise-Resilient Diffusion Spectral Entropy [55.014926694758195]
  Entropy and mutual information in neural networks provide rich information on the learning process. We leverage data geometry to access the underlying manifold and reliably compute these information-theoretic measures. We show that they form noise-resistant measures of intrinsic dimensionality and relationship strength in high-dimensional simulated data.
  arXiv  Detail & Related papers  (2023-12-04T01:32:42Z)
• Uncertainty Estimation by Fisher Information-based Evidential Deep Learning [61.94125052118442]
  Uncertainty estimation is a key factor that makes deep learning reliable in practical applications. We propose a novel method, Fisher Information-based Evidential Deep Learning ($mathcalI$-EDL) In particular, we introduce Fisher Information Matrix (FIM) to measure the informativeness of evidence carried by each sample, according to which we can dynamically reweight the objective loss terms to make the network more focused on the representation learning of uncertain classes.
  arXiv  Detail & Related papers  (2023-03-03T16:12:59Z)
• On the Effective Usage of Priors in RSS-based Localization [56.68864078417909]
  We propose a Received Signal Strength (RSS) fingerprint and convolutional neural network-based algorithm, LocUNet. In this paper, we study the localization problem in dense urban settings. We first recognize LocUNet's ability to learn the underlying prior distribution of the Rx position or Rx and transmitter (Tx) association preferences from the training data, and attribute its high performance to these.
  arXiv  Detail & Related papers  (2022-11-28T00:31:02Z)
• Likelihood-Free Parameter Estimation with Neural Bayes Estimators [0.0]
  Neural point estimators are neural networks that map data to parameter point estimates. We aim to increase the awareness of statisticians to this relatively new inferential tool, and to facilitate its adoption by providing user-friendly open-source software.
  arXiv  Detail & Related papers  (2022-08-27T06:58:16Z)
• Look beyond labels: Incorporating functional summary information in Bayesian neural networks [11.874130244353253]
  We present a simple approach to incorporate summary information about the predicted probability. The available summary information is incorporated as augmented data and modeled with a Dirichlet process. We show how the method can inform the model about task difficulty or class imbalance.
  arXiv  Detail & Related papers  (2022-07-04T07:06:45Z)
• A Perspective on Neural Capacity Estimation: Viability and Reliability [9.251773744318118]
  We study the performance of neural mutual information estimators (NMIE) proposed in the literature when applied to the capacity estimation problem. For the NMIE above, capacity estimation relies on two deep neural networks (DNN) We benchmark these NMIE in three scenarios: (i) AWGN channel capacity estimation and (ii) channels with unknown capacity and continuous inputs.
  arXiv  Detail & Related papers  (2022-03-22T14:55:31Z)
• A Variational Bayesian Approach to Learning Latent Variables for Acoustic Knowledge Transfer [55.20627066525205]
  We propose a variational Bayesian (VB) approach to learning distributions of latent variables in deep neural network (DNN) models. Our proposed VB approach can obtain good improvements on target devices, and consistently outperforms 13 state-of-the-art knowledge transfer algorithms.
  arXiv  Detail & Related papers  (2021-10-16T15:54:01Z)
• Efficient training of lightweight neural networks using Online Self-Acquired Knowledge Distillation [51.66271681532262]
  Online Self-Acquired Knowledge Distillation (OSAKD) is proposed, aiming to improve the performance of any deep neural model in an online manner. We utilize k-nn non-parametric density estimation technique for estimating the unknown probability distributions of the data samples in the output feature space.
  arXiv  Detail & Related papers  (2021-08-26T14:01:04Z)
• MET: Multimodal Perception of Engagement for Telehealth [52.54282887530756]
  We present MET, a learning-based algorithm for perceiving a human's level of engagement from videos. We release a new dataset, MEDICA, for mental health patient engagement detection.
  arXiv  Detail & Related papers  (2020-11-17T15:18:38Z)
• Neural Approximate Sufficient Statistics for Implicit Models [34.44047460667847]
  We frame the task of constructing sufficient statistics as learning mutual information maximizing representations of the data with the help of deep neural networks. We apply our approach to both traditional approximate Bayesian computation and recent neural likelihood methods, boosting their performance on a range of tasks.
  arXiv  Detail & Related papers  (2020-10-20T07:11:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.