A unifying view of contrastive learning, importance sampling, and bridge sampling for energy-based models

• URL: http://arxiv.org/abs/2604.08116v1
• Date: Thu, 09 Apr 2026 11:36:04 GMT
• Title: A unifying view of contrastive learning, importance sampling, and bridge sampling for energy-based models
• Abstract summary: We provide a unified framework that connects noise contrastive estimation (NCE), reverse logistic regression (RLR), multiple importance sampling (MIS), and bridge sampling.<n>This unified perspective clarifies relationships among existing methods and enables the development of new estimators.<n>This study helps elucidate the success of NCE in terms of its flexibility and robustness, while also identifying scenarios in which its performance can be further improved.
• Score: 1.9459606412374646
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the last decades, energy-based models (EBMs) have become an important class of probabilistic models in which a component of the likelihood is intractable and therefore cannot be evaluated explicitly. Consequently, parameter estimation in EBMs is challenging for conventional inference methods. In this work, we provide a unified framework that connects noise contrastive estimation (NCE), reverse logistic regression (RLR), multiple importance sampling (MIS), and bridge sampling within the context of EBMs. We further show that these methods are equivalent under specific conditions. This unified perspective clarifies relationships among existing methods and enables the development of new estimators, with the potential to improve statistical and computational efficiency. Furthermore, this study helps elucidate the success of NCE in terms of its flexibility and robustness, while also identifying scenarios in which its performance can be further improved. Hence, rather than being a purely descriptive review, this work offers a unifying perspective and additional methodological contributions. The MATLAB code used in the numerical experiments is also made freely available to support the reproducibility of the results.

Related papers

• Interpretable and Fair Generalized Additive Neural Networks via Multi-objective Learning [52.66717986145448]
  Interpretability and fairness are two of the most emphasized dimensions in trustworthy artificial intelligence (AI)<n>This paper focuses on neural network (NN)-based generalized additive models (GAMs), a class of self-interpretable models.
  arXiv  Detail & Related papers  (2026-09-05T07:29:42Z)
• Benchmark of Likelihood-Free Inference Methods based on Neural and Optimal Transport Approaches [0.0]
  We investigate how the performance of likelihood-free inference methods depends on structural features such as heavy-tails or discreteness.<n>We consider four approaches: MLE, NBE, EOT and AW--NBE and evaluate them using simulations.
  arXiv  Detail & Related papers  (2026-05-28T19:53:30Z)
• First-Order Efficiency for Probabilistic Value Estimation via A Statistical Viewpoint [18.696678371045216]
  We propose an efficiency-Aware Surrogate-adjusted Estimator (EASE) that directly chooses the sampling law and surrogate to minimize the first-order MSE.<n>We demonstrate that EASE consistently outperforms state-of-the-art estimators for various probabilistic values.
  arXiv  Detail & Related papers  (2026-05-04T17:02:17Z)
• CTRLS: Chain-of-Thought Reasoning via Latent State-Transition [57.51370433303236]
  Chain-of-thought (CoT) reasoning enables large language models to break down complex problems into interpretable intermediate steps.<n>We introduce groundingS, a framework that formulates CoT reasoning as a Markov decision process (MDP) with latent state transitions.<n>We show improvements in reasoning accuracy, diversity, and exploration efficiency across benchmark reasoning tasks.
  arXiv  Detail & Related papers  (2025-07-10T21:32:18Z)
• Confidence in Large Language Model Evaluation: A Bayesian Approach to Limited-Sample Challenges [13.526258635654882]
  This study introduces a Bayesian approach for large language models (LLMs) capability assessment.<n>We treat model capabilities as latent variables and leverage a curated query set to induce discriminative responses.<n> Experimental evaluations with GPT-series models demonstrate that the proposed method achieves superior discrimination compared to conventional evaluation methods.
  arXiv  Detail & Related papers  (2025-04-30T04:24:50Z)
• BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning [78.63421517563056]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks.<n>We present a unified probabilistic framework that formalizes LLM reasoning through a novel graphical model.<n>We introduce the Bootstrapping Reinforced Thinking Process (BRiTE) algorithm, which works in two steps.
  arXiv  Detail & Related papers  (2025-01-31T02:39:07Z)
• Exogenous Matching: Learning Good Proposals for Tractable Counterfactual Estimation [1.9662978733004601]
  We propose an importance sampling method for tractable and efficient estimation of counterfactual expressions.<n>By minimizing a common upper bound of counterfactual estimators, we transform the variance minimization problem into a conditional distribution learning problem.<n>We validate the theoretical results through experiments under various types and settings of Structural Causal Models (SCMs) and demonstrate the outperformance on counterfactual estimation tasks.
  arXiv  Detail & Related papers  (2024-10-17T03:08:28Z)
• Latent Variable Representation for Reinforcement Learning [131.03944557979725]
  It remains unclear theoretically and empirically how latent variable models may facilitate learning, planning, and exploration to improve the sample efficiency of model-based reinforcement learning. We provide a representation view of the latent variable models for state-action value functions, which allows both tractable variational learning algorithm and effective implementation of the optimism/pessimism principle. In particular, we propose a computationally efficient planning algorithm with UCB exploration by incorporating kernel embeddings of latent variable models.
  arXiv  Detail & Related papers  (2022-12-17T00:26:31Z)
• A Unified Contrastive Energy-based Model for Understanding the Generative Ability of Adversarial Training [64.71254710803368]
  Adversarial Training (AT) is an effective approach to enhance the robustness of deep neural networks. We demystify this phenomenon by developing a unified probabilistic framework, called Contrastive Energy-based Models (CEM) We propose a principled method to develop adversarial learning and sampling methods.
  arXiv  Detail & Related papers  (2022-03-25T05:33:34Z)
• Surrogate Likelihoods for Variational Annealed Importance Sampling [11.144915453864854]
  We introduce a surrogate likelihood that can be learned jointly with other variational parameters. We show that our method performs well in practice and that it is well-suited for black-box inference in probabilistic programming frameworks.
  arXiv  Detail & Related papers  (2021-12-22T19:49:45Z)
• MINIMALIST: Mutual INformatIon Maximization for Amortized Likelihood Inference from Sampled Trajectories [61.3299263929289]
  Simulation-based inference enables learning the parameters of a model even when its likelihood cannot be computed in practice. One class of methods uses data simulated with different parameters to infer an amortized estimator for the likelihood-to-evidence ratio. We show that this approach can be formulated in terms of mutual information between model parameters and simulated data.
  arXiv  Detail & Related papers  (2021-06-03T12:59:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.