Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations

• URL: http://arxiv.org/abs/2303.03307v2
• Date: Sun, 3 Dec 2023 21:35:35 GMT
• Title: Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations
• Authors: Thomas Yerxa, Yilun Kuang, Eero Simoncelli, SueYeon Chung
• Abstract summary: The efficient coding hypothesis proposes that the response properties of sensory systems are adapted to the statistics of their inputs. While elegant, information theoretic properties are notoriously difficult to measure in practical settings or to employ as objective functions in optimization. Here we outline the assumptions that allow manifold capacity to be optimized directly, yielding Maximum Manifold Capacity Representations (MMCR)
• Score: 4.666056064419346
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The efficient coding hypothesis proposes that the response properties of sensory systems are adapted to the statistics of their inputs such that they capture maximal information about the environment, subject to biological constraints. While elegant, information theoretic properties are notoriously difficult to measure in practical settings or to employ as objective functions in optimization. This difficulty has necessitated that computational models designed to test the hypothesis employ several different information metrics ranging from approximations and lower bounds to proxy measures like reconstruction error. Recent theoretical advances have characterized a novel and ecologically relevant efficiency metric, the manifold capacity, which is the number of object categories that may be represented in a linearly separable fashion. However, calculating manifold capacity is a computationally intensive iterative procedure that until now has precluded its use as an objective. Here we outline the simplifying assumptions that allow manifold capacity to be optimized directly, yielding Maximum Manifold Capacity Representations (MMCR). The resulting method is closely related to and inspired by advances in the field of self supervised learning (SSL), and we demonstrate that MMCRs are competitive with state of the art results on standard SSL benchmarks. Empirical analyses reveal differences between MMCRs and representations learned by other SSL frameworks, and suggest a mechanism by which manifold compression gives rise to class separability. Finally we evaluate a set of SSL methods on a suite of neural predictivity benchmarks, and find MMCRs are higly competitive as models of the ventral stream.

Related papers

• Recursive Learning of Asymptotic Variational Objectives [49.69399307452126]
  General state-space models (SSMs) are widely used in statistical machine learning and are among the most classical generative models for sequential time-series data. Online sequential IWAE (OSIWAE) allows for online learning of both model parameters and a Markovian recognition model for inferring latent states. This approach is more theoretically well-founded than recently proposed online variational SMC methods.
  arXiv  Detail & Related papers  (2024-11-04T16:12:37Z)
• Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models [66.1595537904019]
  Large language models (LLMs) can act as gradient priors in a zero-shot setting. We introduce LM-GC, a novel method that integrates LLMs with arithmetic coding.
  arXiv  Detail & Related papers  (2024-09-26T13:38:33Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Maximum Manifold Capacity Representations in State Representation Learning [8.938418994111716]
  manifold-based self-supervised learning (SSL) builds on the manifold hypothesis. DeepInfomax with an unbalanced atlas (DIM-UA) has emerged as a powerful tool. MMCR presents a new frontier for SSL by optimizing class separability via manifold compression. We present an innovative integration of MMCR into existing SSL methods, incorporating a discerning regularization strategy.
  arXiv  Detail & Related papers  (2024-05-22T17:19:30Z)
• Latent Semantic Consensus For Deterministic Geometric Model Fitting [109.44565542031384]
  We propose an effective method called Latent Semantic Consensus (LSC) LSC formulates the model fitting problem into two latent semantic spaces based on data points and model hypotheses. LSC is able to provide consistent and reliable solutions within only a few milliseconds for general multi-structural model fitting.
  arXiv  Detail & Related papers  (2024-03-11T05:35:38Z)
• Synergistic eigenanalysis of covariance and Hessian matrices for enhanced binary classification [72.77513633290056]
  We present a novel approach that combines the eigenanalysis of a covariance matrix evaluated on a training set with a Hessian matrix evaluated on a deep learning model. Our method captures intricate patterns and relationships, enhancing classification performance.
  arXiv  Detail & Related papers  (2024-02-14T16:10:42Z)
• Online Variational Sequential Monte Carlo [49.97673761305336]
  We build upon the variational sequential Monte Carlo (VSMC) method, which provides computationally efficient and accurate model parameter estimation and Bayesian latent-state inference. Online VSMC is capable of performing efficiently, entirely on-the-fly, both parameter estimation and particle proposal adaptation.
  arXiv  Detail & Related papers  (2023-12-19T21:45:38Z)
• Extension of Transformational Machine Learning: Classification Problems [0.0]
  This study explores the application and performance of Transformational Machine Learning (TML) in drug discovery. TML, a meta learning algorithm, excels in exploiting common attributes across various domains. The drug discovery process, which is complex and time-consuming, can benefit greatly from the enhanced prediction accuracy.
  arXiv  Detail & Related papers  (2023-08-07T07:34:18Z)
• Self-learning locally-optimal hypertuning using maximum entropy, and comparison of machine learning approaches for estimating fatigue life in composite materials [0.0]
  We develop an ML nearest-neighbors-alike algorithm based on the principle of maximum entropy to predict fatigue damage. The predictions achieve a good level of accuracy, similar to other ML algorithms.
  arXiv  Detail & Related papers  (2022-10-19T12:20:07Z)
• Autoregressive Hidden Markov Models with partial knowledge on latent space applied to aero-engines prognostics [2.179313476241343]
  This paper describes an Autoregressive Partially-hidden Markov model (ARPHMM) for fault detection and prognostics of equipments based on sensors' data. We show how to apply this model to estimate the remaining useful life based on health indicators.
  arXiv  Detail & Related papers  (2021-05-01T10:23:22Z)
• Revisiting minimum description length complexity in overparameterized models [38.21167656112762]
  We provide an extensive theoretical characterization of MDL-COMP for linear models and kernel methods. For kernel methods, we show that MDL-COMP informs minimax in-sample error, and can decrease as the dimensionality of the input increases. We also prove that MDL-COMP bounds the in-sample mean squared error (MSE)
  arXiv  Detail & Related papers  (2020-06-17T22:45:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.