Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

• URL: http://arxiv.org/abs/2603.12228v1
• Date: Thu, 12 Mar 2026 17:49:30 GMT
• Title: Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights
• Abstract summary: We show that in small models such expert solutions occupy a negligible fraction of the volume of this distribution.<n>We explore a simple, fully parallel post-training method that samples $N$ parameter perturbations at random, selects the top $K$, and ensembles predictions via majority vote.
• Score: 32.68928697721277
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pretraining produces a learned parameter vector that is typically treated as a starting point for further iterative adaptation. In this work, we instead view the outcome of pretraining as a distribution over parameter vectors, whose support already contains task-specific experts. We show that in small models such expert solutions occupy a negligible fraction of the volume of this distribution, making their discovery reliant on structured optimization methods such as gradient descent. In contrast, in large, well-pretrained models the density of task-experts increases dramatically, so that diverse, task-improving specialists populate a substantial fraction of the neighborhood around the pretrained weights. Motivated by this perspective, we explore a simple, fully parallel post-training method that samples $N$ parameter perturbations at random, selects the top $K$, and ensembles predictions via majority vote. Despite its simplicity, this approach is competitive with standard post-training methods such as PPO, GRPO, and ES for contemporary large-scale models.

Related papers

• PACT: Preserving Anchored Cores in Task-vectors for Model Merging [68.52455853496585]
  Model merging aims to combine multiple task-specific fine-tuned models into a single multi-task model.<n>Most existing model merging approaches follow the Task Arithmetic paradigm.<n>We propose PACT, which preserves the anchored task-specific cores (i.e., LBW dimensions) within task vectors by aligning their complements with the subspace of the pre-trained weights.
  arXiv  Detail & Related papers  (2026-06-17T02:48:35Z)
• Evolution-Inspired Sample Competition for Deep Neural Network Optimization [19.75458355825581]
  We present textitNatural Selection (NS), a novel evolution-inspired optimization method that explicitly incorporates competitive interactions into deep network training.<n>NS estimates the competitive status of each sample in a group-wise context and uses it to adaptively regulate its training contribution.<n>NS is compatible with diverse network architectures and does not depend on task-specific assumptions.
  arXiv  Detail & Related papers  (2026-04-14T10:48:01Z)
• Task Addition and Weight Disentanglement in Closed-Vocabulary Models [75.01322212415435]
  Task arithmetic has emerged as a promising method for editing pre-trained textitopen-vocabulary models.<n>In this paper, we study task addition in closed-vocabulary image classification models.<n>We find that pre-trained vision transformers can also be edited with task arithmetic.
  arXiv  Detail & Related papers  (2025-11-18T15:12:21Z)
• Fisher meets Feynman: score-based variational inference with a product of experts [28.07579363570697]
  We introduce a highly expressive yet distinctly tractable family for black-box variational inference (BBVI)<n>Each member of this family is a weighted product of experts (PoE)<n>We show how to do this by reformulating these products of experts as latent variable models with auxiliary Dirichlet random variables.
  arXiv  Detail & Related papers  (2025-10-24T16:02:18Z)
• Reasoning with Sampling: Your Base Model is Smarter Than You Think [52.639108524651846]
  We propose a simple iterative sampling algorithm leveraging the base models' own likelihoods.<n>We show that our algorithm offers substantial boosts in reasoning that nearly match and even outperform those from RL.<n>Our method does not require training, curated datasets, or a verifier.
  arXiv  Detail & Related papers  (2025-10-16T17:18:11Z)
• Test-Time Alignment via Hypothesis Reweighting [56.71167047381817]
  Large pretrained models often struggle with underspecified tasks.<n>We propose a novel framework to address the challenge of aligning models to test-time user intent.
  arXiv  Detail & Related papers  (2024-12-11T23:02:26Z)
• Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning [78.72226641279863]
  Sparse Mixture of Expert (SMoE) models have emerged as a scalable alternative to dense models in language modeling. Our research explores task-specific model pruning to inform decisions about designing SMoE architectures. We introduce an adaptive task-aware pruning technique UNCURL to reduce the number of experts per MoE layer in an offline manner post-training.
  arXiv  Detail & Related papers  (2024-09-02T22:35:03Z)
• A Diffusion Model Framework for Unsupervised Neural Combinatorial Optimization [23.972397132797116]
  Current deep learning approaches rely on generative models that yield exact sample likelihoods.<n>This work introduces a method that lifts this restriction and opens the possibility to employ highly expressive latent variable models.<n>We experimentally validate our approach in data-free Combinatorial Optimization and demonstrate that our method achieves a new state-of-the-art on a wide range of benchmark problems.
  arXiv  Detail & Related papers  (2024-06-03T17:55:02Z)
• Training Discrete Deep Generative Models via Gapped Straight-Through Estimator [72.71398034617607]
  We propose a Gapped Straight-Through ( GST) estimator to reduce the variance without incurring resampling overhead. This estimator is inspired by the essential properties of Straight-Through Gumbel-Softmax. Experiments demonstrate that the proposed GST estimator enjoys better performance compared to strong baselines on two discrete deep generative modeling tasks.
  arXiv  Detail & Related papers  (2022-06-15T01:46:05Z)
• Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
  Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios. We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
  arXiv  Detail & Related papers  (2022-04-13T12:43:12Z)
• Deep Ensembles for Low-Data Transfer Learning [21.578470914935938]
  We study different ways of creating ensembles from pre-trained models. We show that the nature of pre-training itself is a performant source of diversity. We propose a practical algorithm that efficiently identifies a subset of pre-trained models for any downstream dataset.
  arXiv  Detail & Related papers  (2020-10-14T07:59:00Z)
• Robust Sampling in Deep Learning [62.997667081978825]
  Deep learning requires regularization mechanisms to reduce overfitting and improve generalization. We address this problem by a new regularization method based on distributional robust optimization. During the training, the selection of samples is done according to their accuracy in such a way that the worst performed samples are the ones that contribute the most in the optimization.
  arXiv  Detail & Related papers  (2020-06-04T09:46:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.