Affinity Is Not Enough: Recovering the Free Energy Principle in Mixture-of-Experts

• URL: http://arxiv.org/abs/2605.00604v1
• Date: Fri, 01 May 2026 12:18:55 GMT
• Title: Affinity Is Not Enough: Recovering the Free Energy Principle in Mixture-of-Experts
• Abstract summary: Sfinity MoE fails at domain transitions, where the current token belongs to one distribution and the next to another.<n>Three lightweight gate modifications raise this to 0.748 +/- 0.002 (124x)<n>We show that beta-routing reduces transition-step B from 6.56 +/- 0.01 to 4.01 +/- 0.15 (beta-MoE)
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse MoE routing fails at domain transitions, where the current token belongs to one distribution and the next to another. In a controlled experiment (4 experts, 5 seeds), standard affinity routing assigns only 0.006 +/- 0.001 probability to the correct expert at the transition. Three lightweight gate modifications raise this to 0.748 +/- 0.002 (124x), cutting experts needed for 99% coverage from infeasible to a small constant: temporal memory (beta), a per-expert LIF membrane potential accumulating routing context across tokens; precision-weighted gating (Pi), a per-expert inverse variance of recent prediction error, yielding 31x contrast between reliable and unreliable experts; and anticipatory routing, a next-state predictor conditioned on the beta-accumulated hidden state. The mechanisms draw from Friston's Free Energy Principle and use LIF dynamics from spiking neural networks. An ablation across all 2^3 subsets reveals a super-additive beta x Ant interaction: anticipation alone gives nothing (+0.000 +/- 0.001); beta alone gives modest gain (+0.295 +/- 0.013); combined they close 75% of the oracle gap (+0.741 +/- 0.002, exceeding the sum by +0.446 +/- 0.014). This is structural: a stateless predictor cannot detect approaching transitions because pre-transition tokens are distributionally identical to within-domain tokens. In a character-level MoE LM (5 seeds), beta-routing reduces transition-step BPC from 6.56 +/- 0.01 (Standard) to 4.01 +/- 0.15 (beta-MoE); the beta + Ant gate places 0.86 +/- 0.02 probability on the correct domain expert before that domain appears in input, vs 0.42 +/- 0.12 for Standard MoE. Reference implementations (~200 lines each): https://github.com/russellwmy/affinity-is-not-enough

Related papers

• Haar-Bayesian Pure-State Prediction under Relative-Entropy Loss: Arbitrary-Effect Reduction and Global Optimality [42.694152897125726]
  We study Haar-Bayesian prediction of one unmeasured copy of an unknown finite-dimensional pure quantum state.<n>Performance is evaluated by quantum relative entropy.
  arXiv  Detail & Related papers  (2026-09-09T11:53:28Z)
• Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk [0.0]
  Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters.<n>A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale.<n>We build such a system end to end and test whether the second half of the promise holds.
  arXiv  Detail & Related papers  (2026-08-08T13:22:14Z)
• Multi-Horizon Consistency as Geometry: When Latent Dynamics Contract, and When They Do Not [0.3058685580689604]
  Multi-step latent agreement is a common training knob in video predictors and world models.<n>We measure an empirical expansion proxy L20,q95 together with horizon-20 prediction error E20.<n>Four of six seeds cross L1 at=0.8. The same loss does not produce population L1 on action-v1 or CartPole-v1, nor on KTH video, even when E20 improves.
  arXiv  Detail & Related papers  (2026-07-21T23:50:10Z)
• Operational Proto-Introspection in Looped Language Models: Process-Quality Taps, Executable Branching, and the Readout-Control Boundary [0.0]
  Can a language model read the quality of its ongoing computation?<n>Can an external intervention turn that readout into better outcomes?<n>We test both questions in a frozen 2.6B looped transformer, Ouro-RLTT.
  arXiv  Detail & Related papers  (2026-07-20T22:40:36Z)
• Reliable Extraction of Clinical Follow-Up Instructions: A Hybrid Neural-Symbolic Pipeline [43.0484058393522]
  We test a hybrid neural-symbolic pipeline against direct generation.<n>We evaluate on a 2,000-note synthetic outpatient corpus with action-disjoint splits.
  arXiv  Detail & Related papers  (2026-05-26T05:14:33Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs [52.709361620508595]
  ListOPD brings a student to in-domain with an 8B-SFT baseline at one-fifth the parameters.<n>On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval test, and a small-clip cross-prediction--fall within their locked prediction windows--with the small-clip value matching the closed-form prediction below grid resolution.
  arXiv  Detail & Related papers  (2026-05-09T06:48:00Z)
• PHANTOM: Polymorphic Honeytoken Adaptation with Narrative-Tailored Organisational Mimicry [0.2538209532048867]
  Honeytokens are decoy digital assets planted to detect and unauthorised attribute access.<n>We introduce PHANTOM, a framework that generates contextually convincing honeytokens by encoding organisation-specific knowledge.<n>Results are reproduced without external API calls, making the pipeline fully deployable in air-gapped environments.
  arXiv  Detail & Related papers  (2026-05-04T17:14:13Z)
• Lost in State Space: Probing Frozen Mamba Representations [0.5156484100374058]
  Mamba's recurrent state h_t is a compressed summary of every token seen so far.<n>If we extract token-level outputs y_t at fixed patch boundaries, we obtain semantic sentence summaries for free.<n>We compare four strategies for extracting frozen sentence representations from a pretrained Mamba-130M backbone.
  arXiv  Detail & Related papers  (2026-04-30T21:35:08Z)
• THEIA: Learning Complete Kleene Three-Valued Logic in a Pure-Neural Modular Architecture [0.0]
  THEIA is a 2.75M modular neural architecture that learns the complete Kleene three-valued logic (K3) truth table from task data without external symbolic inference or hand-encoded K3 gate primitives.<n> Transformer baselines also reach >99% on all 39 rules, and flats match THEIA on Phase-1 accuracy within 0.04pp.
  arXiv  Detail & Related papers  (2026-04-13T10:44:15Z)
• Next-Token Prediction and Regret Minimization [39.73178505655866]
  We consider the question of how to employ next-token prediction algorithms in adversarial online decision-making environments.<n>We show that although not every distribution $mathcalD$ is a low-regret distribution, every distribution $mathcalD$ is exponentially close to one low-regret distribution.
  arXiv  Detail & Related papers  (2026-03-30T14:34:41Z)
• Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning [61.07540493350384]
  Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth and the teacher's own predictions.<n>We show that for any prediction risk, the optimally mixed student improves upon the ridge teacher for every regularization level.<n>We propose a consistent one-shot tuning method to estimate $star$ without grid search, sample splitting, or refitting.
  arXiv  Detail & Related papers  (2026-02-19T17:21:15Z)
• Unregularized Linear Convergence in Zero-Sum Game from Preference Feedback [50.89125374999765]
  We provide the first convergence guarantee for Optimistic Multiplicative Weights Update ($mathtOMWU$) in NLHF.<n>Our analysis identifies a novel marginal convergence behavior, where the probability of rarely played actions grows exponentially from exponentially small values.
  arXiv  Detail & Related papers  (2025-12-31T12:08:29Z)
• Scale-Invariant Regret Matching and Online Learning with Optimal Convergence: Bridging Theory and Practice in Zero-Sum Games [60.871651115241406]
  A considerable chasm has been looming for decades between theory and practice in zero-sum game solving through first-order methods.<n>We propose a new scale-invariance and parameter-free variant of PRM$+$, which we call IREG-PRM$+$.<n>We show that it achieves $T-1/2$ best-iterate and $T-1$ optimal convergence guarantees, while also being on par with PRM$+$ on benchmark games.
  arXiv  Detail & Related papers  (2025-10-06T00:33:20Z)
• Quantile Advantage Estimation for Entropy-Safe Reasoning [44.192277495613695]
  Reinforcement Learning with Verifiable Rewards (RLVR) strengthens LLM reasoning, but training often oscillates between entropy collapse and entropy explosion<n>We trace both hazards to the mean baseline used in value-free RL, which improperly penalizes negative-advantage samples under reward outliers.<n>We propose Quantile Advantage Estimation (QAE), replacing the mean with a group-wise K-quantile baseline.
  arXiv  Detail & Related papers  (2025-09-26T17:37:52Z)
• Kourkoutas-Beta: A Sunspike-Driven Adam Optimizer with Desert Flair [0.0]
  Transformer neural networks are increasingly used for physics-based problems.<n>In data-driven PDE surrogates, training samples from varying boundary and initial conditions can cause erratic losses and spiky gradients.<n>We introduce Kourkoutas-Beta, an Adam-style discount where the fixed second-moment beta2 is replaced by a layer-wise dynamic value.
  arXiv  Detail & Related papers  (2025-08-18T15:16:54Z)
• Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulation [89.88214896713846]
  STA score considers two critical factors: temporal redundancy and semantic importance. We apply the STA module to off-the-shelf video Transformers and Videowins. Results: Kinetics-400 and Something-Something V2 achieve 30% overshelf reduction with a negligible 0.2% accuracy drop.
  arXiv  Detail & Related papers  (2023-08-08T19:38:15Z)
• Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networks [74.68583356645276]
  In deep learning, mixture-of-experts (MoE) activates one or few experts (sub-networks) on a per-sample or per-token basis. We show for the first time that pMoE provably reduces the required number of training samples to achieve desirable generalization.
  arXiv  Detail & Related papers  (2023-06-07T00:16:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.