Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation

• URL: http://arxiv.org/abs/2607.23860v1
• Date: Sun, 26 Jul 2026 22:06:42 GMT
• Title: Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation
• Abstract summary: $$N-Ens is a normalisation-based implicit ensemble that treats each member as a task in a multi-task architecture.<n>Because only normalisation layers are replicated, the mechanism can wrap convolutional and transformer backbones alike.
• Score: 0.9668407688201359
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep ensembles provide the most reliable uncertainty estimates in deep learning, but their cost grows linearly with the number of members. Implicit ensembles lower this cost by sharing a single backbone across members. Member diversity is a primary determinant of ensemble quality, yet no implicit ensemble can shape it during training; existing methods fix it at initialisation or build it into the architecture. We introduce $σ$N-Ens, a normalisation-based implicit ensemble that treats each member as a task in a multi-task architecture and modulates the shared backbone through sigmoid-bounded scalers. We also introduce a softmax-temperature regulariser, which shapes the equilibrium level of sharing between members and traces the accuracy-calibration frontier. Because only normalisation layers are replicated, the mechanism can wrap convolutional and transformer backbones alike, also allowing pretrained models to be adapted through a short fine-tune. We frame the epistemic uncertainty such an ensemble expresses as modulation uncertainty, and explain why its calibration holds under input corruption, and why its out-of-distribution detection is weaker. Our method is evaluated across ResNets and transformers on CIFAR-10/100, ImageNet and SST-2. $σ$N-Ens matches or outperforms deep ensembles at a fraction of their parameter cost, scales with ensemble size where partitioning methods collapse, and maintains calibration under distribution shift.

Related papers

• Simplex Relaxation for Discrete Diffusion [58.33696160279345]
  We introduce Simplax, an exact Dirichlet--categorical augmentation that couples each corrupted categorical state with an auxiliary simplex-valued variable.<n>Simplax improves the generative perplexity--entropy tradeoff on unconditional OpenWebText generation.
  arXiv  Detail & Related papers  (2026-08-11T07:58:17Z)
• DP-Splat: Bayesian Nonparametric Complexity Control for Gaussian Splatting [0.0]
  3D Splatting represents finite mixtures of anisotropic Gaussians whose number of components $K$ is set by density control or user caps.<n>We replace the symmetric symmetric Dirichlet over mixture weights with a truncated stick-breaking Dirichlet-process prior.<n>We give an exact monotonicity guarantee, a rigorous truncation-error bound correcting an anti-conservative large-error approximation in common use.
  arXiv  Detail & Related papers  (2026-07-12T20:31:15Z)
• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• Scaling Autoregressive Models for Lattice Thermodynamics [5.904448014233719]
  Markov-chain Monte Carlo sampling suffers from slow convergence and critical slowing down near phase transitions.<n>Existing autoregressive models (ARMs) generate configurations in a fixed sequential order and incur high memory and training costs.<n>Here, we develop a framework combining any-order ARMs, which generate configurations flexibly by conditioning on any known subset of lattice sites, with models (MAMs) which approximate the probability of any partial configuration in a single forward pass.
  arXiv  Detail & Related papers  (2026-03-16T01:03:59Z)
• Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs [55.77845440440496]
  Push-based decentralized communication enables optimization over communication networks, where information exchange may be asymmetric.<n>We develop a unified uniform-stability framework for the Gradient Push (SGP) algorithm.<n>A key technical ingredient is an imbalance-aware generalization bound through two quantities.
  arXiv  Detail & Related papers  (2026-02-24T05:32:03Z)
• Making Foundation Models Probabilistic via Singular Value Ensembles [56.4174499669573]
  Foundation models have become a dominant paradigm in machine learning, achieving remarkable performance across diverse tasks through large-scale pretraining.<n>Standard approach to quantifying uncertainty, training an ensemble of independent models, incurs prohibitive computational costs that scale linearly with ensemble size.<n>We propose Singular Value Ensemble (SVE), a parameter-efficient implicit ensemble method that builds on a simple, but powerful core assumption.<n>We show that SVE uncertainty quantification achieves comparable to explicit deep ensembles while increasing the parameter count of the base model by less than 1%.
  arXiv  Detail & Related papers  (2026-01-29T18:07:18Z)
• Is BatchEnsemble a Single Model? On Calibration and Diversity of Efficient Ensembles [2.957223821964636]
  BatchEnsemble aims to deliver ensemble-like uncertainty (EU) EU at far lower parameter and memory cost.<n>We show that BatchEnsemble not only underperforms Deep Ensembles but closely tracks a single model baseline.
  arXiv  Detail & Related papers  (2026-01-23T17:50:50Z)
• UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation [104.59740403500132]
  Multi-modal image segmentation faces real-world deployment challenges from incomplete/corrupted modalities degrading performance.<n>We propose a unified modality-relax segmentation network (UniMRSeg) through hierarchical self-supervised compensation (HSSC)<n>Our approach hierarchically bridges representation gaps between complete and incomplete modalities across input, feature and output levels.
  arXiv  Detail & Related papers  (2025-09-19T17:29:25Z)
• Scalable Robust Bayesian Co-Clustering with Compositional ELBOs [2.6756996523251964]
  Co-clustering exploits the duality of instances and features to uncover meaningful groups in both dimensions.<n>We present the first fully variational Co-clustering framework that directly learns row and column clusters in the latent space.<n>Our method not only preserves the advantages of prior Co-clustering approaches but also exceeds them in accuracy and robustness.
  arXiv  Detail & Related papers  (2025-04-05T06:48:05Z)
• LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks [52.46420522934253]
  We introduce LoRA-Ensemble, a parameter-efficient ensembling method for self-attention networks.<n>The method not only outperforms state-of-the-art implicit techniques like BatchEnsemble, but even matches or exceeds the accuracy of an Explicit Ensemble.
  arXiv  Detail & Related papers  (2024-05-23T11:10:32Z)
• Adaptive Correlated Monte Carlo for Contextual Categorical Sequence Generation [77.7420231319632]
  We adapt contextual generation of categorical sequences to a policy gradient estimator, which evaluates a set of correlated Monte Carlo (MC) rollouts for variance control. We also demonstrate the use of correlated MC rollouts for binary-tree softmax models, which reduce the high generation cost in large vocabulary scenarios.
  arXiv  Detail & Related papers  (2019-12-31T03:01:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.