Scalable Expectation Estimation with Subtractive Mixture Models

• URL: http://arxiv.org/abs/2503.21346v1
• Date: Thu, 27 Mar 2025 10:25:03 GMT
• Title: Scalable Expectation Estimation with Subtractive Mixture Models
• Abstract summary: negative mixture models (SMMs) complicate sampling, requiring costly auto-regressive techniques or accept-reject algorithms that do not scale in high dimensions.<n>In this work, we use the difference representation of SMMs to construct an unbiased IS estimator ($DeltatextEx$) that removes the need to sample from the SMM.<n>In our experiments, we show that $DeltatextEx$ can achieve comparable estimation quality to auto-regressive sampling while being considerably faster in MC estimation.
• Score: 11.150301369804664
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Many Monte Carlo (MC) and importance sampling (IS) methods use mixture models (MMs) for their simplicity and ability to capture multimodal distributions. Recently, subtractive mixture models (SMMs), i.e. MMs with negative coefficients, have shown greater expressiveness and success in generative modeling. However, their negative parameters complicate sampling, requiring costly auto-regressive techniques or accept-reject algorithms that do not scale in high dimensions. In this work, we use the difference representation of SMMs to construct an unbiased IS estimator ($\Delta\text{Ex}$) that removes the need to sample from the SMM, enabling high-dimensional expectation estimation with SMMs. In our experiments, we show that $\Delta\text{Ex}$ can achieve comparable estimation quality to auto-regressive sampling while being considerably faster in MC estimation. Moreover, we conduct initial experiments with $\Delta\text{Ex}$ using hand-crafted proposals, gaining first insights into how to construct safe proposals for $\Delta\text{Ex}$.

Related papers

• How to Approximate Inference with Subtractive Mixture Models [35.41848896315461]
  We study how to effectively use subtractive mixture models (SMMs) for approximate inference settings such as variational inference (VI) and importance sampling (IS)<n>We discuss the additional challenges in estimation stability and learning efficiency that they carry and propose ways to overcome them.
  arXiv  Detail & Related papers  (2026-04-17T21:28:24Z)
• Model Selection and Parameter Estimation of Multi-dimensional Gaussian Mixture Model [5.917400794867873]
  We study the problem of learning multi-dimensional Gaussian Mixture Models (GMMs)<n>We first establish an information-theoretic lower sample required for reliable selection.<n>We then propose a thresholding-based estimation that evaluates the spectral gap of an empirical covariance matrix.
  arXiv  Detail & Related papers  (2026-03-20T05:45:24Z)
• DISCO: Diversifying Sample Condensation for Efficient Model Evaluation [59.01400190971061]
  Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact.<n>We argue that promoting diversity among samples is not essential; what matters is to select samples thatmaximise diversity in model responses.<n>Our method, $textbfDiversifying Sample Condensation (DISCO)$, selects the top-k samples with the greatest model disagreements.
  arXiv  Detail & Related papers  (2025-10-09T08:53:59Z)
• Scalable Importance Sampling in High Dimensions with Low-Rank Mixture Proposals [37.634056981112444]
  Importance sampling is a Monte Carlo technique for efficiently estimating the likelihood of rare events.<n>We propose using mixtures of probabilistic principal component analyzers (MPPCA) as the parametric proposal density for importance sampling methods.<n>We validate our method on three simulated systems, demonstrating consistent gains in sample efficiency and quality of failure distribution characterization.
  arXiv  Detail & Related papers  (2025-05-19T16:44:48Z)
• UncertainSAM: Fast and Efficient Uncertainty Quantification of the Segment Anything Model [19.8785302359805]
  This paper presents a theoretically motivated uncertainty quantification model based on a Bayesian entropy formulation.<n>We use this formulation to train USAM, a lightweight post-hoc UQ method.<n>Our proposed deterministic USAM demonstrates superior predictive capabilities on the SA-V, MOSE, ADE20k, DAVIS, and COCO datasets.
  arXiv  Detail & Related papers  (2025-05-08T08:36:23Z)
• Benchmarking Multi-modal Semantic Segmentation under Sensor Failures: Missing and Noisy Modality Robustness [61.87055159919641]
  Multi-modal semantic segmentation (MMSS) addresses the limitations of single-modality data by integrating complementary information across modalities.<n>Despite notable progress, a significant gap persists between research and real-world deployment due to variability and uncertainty in multi-modal data quality.<n>We introduce a robustness benchmark that evaluates MMSS models under three scenarios: Entire-Missing Modality (EMM), Random-Missing Modality (RMM), and Noisy Modality (NM)
  arXiv  Detail & Related papers  (2025-03-24T08:46:52Z)
• MM-RLHF: The Next Step Forward in Multimodal LLM Alignment [59.536850459059856]
  We introduce MM-RLHF, a dataset containing $mathbf120k$ fine-grained, human-annotated preference comparison pairs.<n>We propose several key innovations to improve the quality of reward models and the efficiency of alignment algorithms.<n>Our approach is rigorously evaluated across $mathbf10$ distinct dimensions and $mathbf27$ benchmarks.
  arXiv  Detail & Related papers  (2025-02-14T18:59:51Z)
• Latent Thought Models with Variational Bayes Inference-Time Computation [52.63299874322121]
  Latent Thought Models (LTMs) incorporate explicit latent thought vectors that follow an explicit prior model in latent space.<n>LTMs demonstrate superior sample and parameter efficiency compared to autoregressive models and discrete diffusion models.
  arXiv  Detail & Related papers  (2025-02-03T17:50:34Z)
• Ranked from Within: Ranking Large Multimodal Models Without Labels [73.96543593298426]
  We show that uncertainty scores derived from softmax distributions provide a robust basis for ranking models across various tasks.<n>This facilitates the ranking of LMMs on unlabeled data, providing a practical approach for selecting models for diverse target domains without requiring manual annotation.
  arXiv  Detail & Related papers  (2024-12-09T13:05:43Z)
• MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation [80.47072100963017]
  We introduce a novel and low-compute algorithm, Model Merging with Amortized Pareto Front (MAP) MAP efficiently identifies a set of scaling coefficients for merging multiple models, reflecting the trade-offs involved. We also introduce Bayesian MAP for scenarios with a relatively low number of tasks and Nested MAP for situations with a high number of tasks, further reducing the computational cost of evaluation.
  arXiv  Detail & Related papers  (2024-06-11T17:55:25Z)
• On Least Square Estimation in Softmax Gating Mixture of Experts [78.3687645289918]
  We investigate the performance of the least squares estimators (LSE) under a deterministic MoE model. We establish a condition called strong identifiability to characterize the convergence behavior of various types of expert functions. Our findings have important practical implications for expert selection.
  arXiv  Detail & Related papers  (2024-02-05T12:31:18Z)
• Efficient Propagation of Uncertainty via Reordering Monte Carlo Samples [0.7087237546722617]
  Uncertainty propagation is a technique to determine model output uncertainties based on the uncertainty in its input variables. In this work, we investigate the hypothesis that while all samples are useful on average, some samples must be more useful than others. We introduce a methodology to adaptively reorder MC samples and show how it results in reduction of computational expense of UP processes.
  arXiv  Detail & Related papers  (2023-02-09T21:28:15Z)
• Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tuning [85.55727213502402]
  We focus on improving the few-shot performance of prompt tuning by transferring knowledge from soft prompts of source tasks. We propose Sample-specific Ensemble of Source Models (SESoM) SESoM learns to adjust the contribution of each source model for each target sample separately when ensembling source model outputs.
  arXiv  Detail & Related papers  (2022-10-23T01:33:16Z)
• Low-variance estimation in the Plackett-Luce model via quasi-Monte Carlo sampling [58.14878401145309]
  We develop a novel approach to producing more sample-efficient estimators of expectations in the PL model. We illustrate our findings both theoretically and empirically using real-world recommendation data from Amazon Music and the Yahoo learning-to-rank challenge.
  arXiv  Detail & Related papers  (2022-05-12T11:15:47Z)
• Bayesian Target-Vector Optimization for Efficient Parameter Reconstruction [0.0]
  We introduce a target-vector optimization scheme that considers all $K$ contributions of the model function and that is specifically suited for parameter reconstruction problems. It also enables to determine accurate uncertainty estimates with very few observations of the actual model function.
  arXiv  Detail & Related papers  (2022-02-23T15:13:32Z)
• MCCE: Monte Carlo sampling of realistic counterfactual explanations [2.156170153103442]
  MCCE is a novel on-manifold, actionable and valid counterfactual explanation method. It generates on-manifold, actionable and valid counterfactuals by modeling the joint distribution of the mutable features. We compare MCCE with a range of state-of-the-art on-manifold counterfactual methods using four well-known data sets.
  arXiv  Detail & Related papers  (2021-11-18T16:40:44Z)
• Sparse MoEs meet Efficient Ensembles [49.313497379189315]
  We study the interplay of two popular classes of such models: ensembles of neural networks and sparse mixture of experts (sparse MoEs) We present Efficient Ensemble of Experts (E$3$), a scalable and simple ensemble of sparse MoEs that takes the best of both classes of models, while using up to 45% fewer FLOPs than a deep ensemble.
  arXiv  Detail & Related papers  (2021-10-07T11:58:35Z)
• Model Mismatch Trade-offs in LMMSE Estimation [0.0]
  We consider a linear minimum mean squared error (LMMSE) estimation framework with model mismatch where the assumed model order is smaller than that of the underlying linear system. By modelling the regressors of the underlying system as random variables, we analyse the average behaviour of the mean squared error (MSE) Our results quantify how the MSE depends on the interplay between the number of samples and the number of parameters in the underlying system and in the assumed model.
  arXiv  Detail & Related papers  (2021-05-25T14:16:45Z)
• Reversible Jump PDMP Samplers for Variable Selection [1.5469452301122175]
  A new class of Monte Carlo algorithms based on piecewise deterministic Markov processes (PDMPs) have recently shown great promise. PDMP samplers can only sample from posterior densities that are differentiable almost everywhere. We show how to develop reversible jump PDMP samplers that can jointly explore the discrete space of models and the continuous space of parameters.
  arXiv  Detail & Related papers  (2020-10-22T14:46:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.