Stable Global Weighting of Flow Mixtures using Simplex Exponential Moving Average

• URL: http://arxiv.org/abs/2607.03809v1
• Date: Sat, 04 Jul 2026 10:31:10 GMT
• Title: Stable Global Weighting of Flow Mixtures using Simplex Exponential Moving Average
• Abstract summary: emphAMFmbox-VImbox-sEMA is a two-stage framework featuring a emphstable global weighting mechanism.<n>It achieves consistent NLL improvements over its predecessor emphAMFmbox-VI and avoids the catastrophic transport failures of single-flow baselines.
• Score: 0.8224695424591678
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Normalising flows provide a powerful variational family for approximate inference, yet individual architectures often fail to generalise across heterogeneous posterior geometries. We revisit mixture-based flow formulations and introduce \emph{AMF\mbox{-}VI\mbox{-}sEMA}, a two-stage framework featuring a \emph{stable global weighting} mechanism based on a \emph{Simplex Exponential Moving Average} (sEMA) update. In Stage~1, a heterogeneous set of experts (\textsc{RealNVP}, \textsc{MAF}, \textsc{RBIG}) are trained independently to specialise in distinct structural regimes. In Stage~2, expert parameters are frozen and global mixture weights are learned through a temperature-controlled softmax of average log-likelihoods, followed by a smooth EMA update on the probability simplex. This design produces a tractable, data-agnostic gating mechanism (without per-sample gating or gradient backpropagation through weights) that adaptively reallocates capacity while avoiding component collapse. We evaluate the framework on ten posterior benchmarks: six canonical 2D synthetic families (Banana, X-Shaped, Bimodal, Multimodal, Two-moons, Rings) and four real/low-dimensional Bayesian targets (BLR, BPR, Weibull, Real-GMM2), with stronger baselines (\textsc{NICE}, \textsc{ResFlow}, and EM-Mixing). Comprehensive evaluation covers NLL, KL divergence, Wasserstein-2 distance, and MMD, together with diagnostics of mixture dynamics, hyperparameter sensitivity, and cross-seed robustness. Empirically, \emph{AMF\mbox{-}VI\mbox{-}sEMA} achieves consistent NLL improvements over its predecessor \emph{AMF\mbox{-}VI} and avoids the catastrophic transport failures of single-flow baselines, while maintaining stable weight trajectories ($N_{\mathrm{eff}}{>}1.4$ on all datasets) with minimal computational overhead.

Related papers

• Deep Skew-t Mixture Models [2.5794915063815664]
  High-dimensional clustering is challenging when component distributions are both heavy-tailed and directionally asymmetric.<n>We propose a hierarchical factor-analytic mixture based on the generalised-hyperbolic skew-$t$ normal mean--variance representation.<n>A shared inverse-gamma mixing variable is propagated along each complete latent pathway, allowing heavy tails and directional asymmetry to be modelled jointly.
  arXiv  Detail & Related papers  (2026-09-01T06:02:11Z)
• LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models [56.999481798138625]
  LimiX-2M is a 2M- parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used benchmarks.<n>We present a unified tokenize-and-route framework for strong Tabular foundation models (TFMs)<n>Results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs.
  arXiv  Detail & Related papers  (2026-06-03T06:07:33Z)
• Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training [56.323119575322146]
  Extra-Merge consistently outperforms standard merging baselines in experiments across GPT-2 and LLaMA families.<n>It yields consistent zero-shot accuracy gains on Pythia-12B downstream tasks and generalizes effectively to the Muon citepjordan2024muon.
  arXiv  Detail & Related papers  (2026-05-26T02:48:34Z)
• OmniISR: A Unified Framework for Centralized and Federated Learning via Intermediate Supervision and Regularization [58.03221830946145]
  We propose OmniISR, a unified framework that fuses pure CL, pure FL, and hybrid CL-FL training modes.<n>We show that OmniISR consistently improves model performance in both centralized and federated paradigms.
  arXiv  Detail & Related papers  (2026-05-19T04:13:27Z)
• GeoFlowVLM: Geometry-Aware Joint Uncertainty for Frozen Vision-Language Embedding [3.0708725114491293]
  We propose textbfGeoFlowVLM as a post-hoc adapter that learns the joint distribution of paired $ell$-normalised dual-encoder VLM embeddings.<n>A consistency result shows that, in the population limit, the trained network exposes the joint flow and both cross-modal conditional flows.<n>We derive two quantities from this single model: a conditional retrieval entropy that quantifies aleatoric ambiguity with a decision-theoretic interpretation via a Fano-type bound, and a marginaltypicality score justified by an exact chain-rule decomposition of the joint NLL
  arXiv  Detail & Related papers  (2026-05-13T11:12:18Z)
• ConsistentRFT: Reducing Visual Hallucinations in Flow-based Reinforcement Fine-Tuning [85.20505958752928]
  Reinforcement Fine-Tuning (RFT) on flow-based models is crucial for preference alignment.<n>RFT often introduce visual hallucinations like over-optimized details and semantic misalignment.<n>This work preliminarily explores why visual hallucinations arise and how to reduce them.
  arXiv  Detail & Related papers  (2026-02-03T11:49:46Z)
• A Fast and Flat Federated Learning Method via Weighted Momentum and Sharpness-Aware Minimization [16.674348270108993]
  We propose a momentum-guided global perturbation from server-aggregated momentum to align clients' SAM directions with the global descent geometry.<n>We provide a non-IID convergence bound that emphexplicitly models the perturbation-induced variance $_2=2+(L)2$ and its dependence on $(S, K, R, N)$ on the theory side.
  arXiv  Detail & Related papers  (2025-11-27T04:02:09Z)
• Sparse by Rule: Probability-Based N:M Pruning for Spiking Neural Networks [66.61171793101872]
  Spiking neural networks (SNNs) promise energy-efficient intelligence via event-driven, sparse computation, but deeper architectures inflate parameters and computational cost, hindering their edge deployment.<n>Recent progress in SNN pruning helps alleviate this burden, yet existing efforts fall into only two families: emphunstructured pruning, which attains high sparsity but is difficult to accelerate on general hardware, and emphstructured pruning, which eases deployment but lack flexibility and often degrades accuracy at matched sparsity.<n>We introduce bfSpikeNM, the first SNN
  arXiv  Detail & Related papers  (2025-11-15T08:23:20Z)
• MTmixAtt: Integrating Mixture-of-Experts with Multi-Mix Attention for Large-Scale Recommendation [8.34766340139746]
  We propose textbfMTmixAtt, a unified Mixture-of-Experts (MoE) architecture with Multi-Mix Attention.<n>The textbfAutoToken module automatically clusters heterogeneous features into semantically coherent tokens, removing the need for human-defined feature groups.<n>Extensive experiments on the industrial TRec dataset from Meituan demonstrate that MTmixAtt consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-10-17T03:50:09Z)
• Adaptive Heterogeneous Mixtures of Normalising Flows for Robust Variational Inference [0.0]
  We propose Adaptive Mixture Flow Variational Inference (AMF-VI)<n>AMF-VI is trained in two stages: (i) sequential expert training of individual flows, and (ii) adaptive global weight estimation via likelihood-driven updates.<n>We evaluate AMF-VI on six canonical posterior families of banana, X-shape, two-moons, rings, a bimodal, and a five-mode mixture.
  arXiv  Detail & Related papers  (2025-10-02T14:25:29Z)
• Global Convergence of Gradient EM for Over-Parameterized Gaussian Mixtures [53.51230405648361]
  We study the dynamics of gradient EM and employ tensor decomposition to characterize the geometric landscape of the likelihood loss.<n>This is the first global convergence and recovery result for EM or gradient EM beyond the special case of $m=2$.
  arXiv  Detail & Related papers  (2025-06-06T23:32:38Z)
• DFedADMM: Dual Constraints Controlled Model Inconsistency for Decentralized Federated Learning [52.83811558753284]
  Decentralized learning (DFL) discards the central server and establishes a decentralized communication network. Existing DFL methods still suffer from two major challenges: local inconsistency and local overfitting.
  arXiv  Detail & Related papers  (2023-08-16T11:22:36Z)
• Gravitational-wave parameter estimation with autoregressive neural network flows [0.0]
  We introduce the use of autoregressive normalizing flows for rapid likelihood-free inference of binary black hole system parameters from gravitational-wave data with deep neural networks. A normalizing flow is an invertible mapping on a sample space that can be used to induce a transformation from a simple probability distribution to a more complex one. We build a more powerful latent variable model by incorporating autoregressive flows within the variational autoencoder framework.
  arXiv  Detail & Related papers  (2020-02-18T15:44:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.