Decomposing Evolutionary Mixture-of-LoRA Architectures: The Routing Lever, the Lifecycle Penalty, and a Substrate-Conditional Boundary

• URL: http://arxiv.org/abs/2605.11153v1
• Date: Mon, 11 May 2026 18:59:11 GMT
• Title: Decomposing Evolutionary Mixture-of-LoRA Architectures: The Routing Lever, the Lifecycle Penalty, and a Substrate-Conditional Boundary
• Abstract summary: We decompose an evolutionary mixtureofLoRA system into three factors -- a router rewrite, a perdomain leave-one-out evaluation scope, and a lifecycle of death plus alpha-blend inheritance plus SVD mutation plus slot reallocation.<n>A synthetic sandbox locates a substrate-conditional regime: evolutionary search on the routing channel is load-bearing only when are pre-aligned to the task; in every other regime tested it underperforms, ties, or actively degrades the solution.
• Score: 3.151184728006369
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We decompose an evolutionary mixture-of-LoRA system on a from-scratch ~150M-parameter widened-D substrate (D=1536, V=32000; D/V approx 0.048; the "widened-1536" substrate) into three factors -- a router rewrite (parallel sigmoid gate with learnable per-adapter floor and bounded temperature anneal, fed post-stack hidden states rather than token-embedding means), a per-domain leave-one-out evaluation scope, and a lifecycle of death plus alpha-blend inheritance plus SVD mutation plus slot reallocation -- and report a 5-of-8 partial 2^3 factorial run at n=3 seeds and 25000 adaptation steps per cell. The attribution chain is sharp on this substrate: the router rewrite carries the entire +0.0426 nat balanced log-PPL improvement (Delta = log PPL_ref - log PPL_test, positive = improvement; t=12.86, p=0.006) attributed to "the full evolutionary system vs the static B3 baseline"; the headline full-system-vs-B3 balanced contrast itself is +0.015 nats, t=1.94, p=0.19 at n=3 and does not clear alpha=0.05. The per-domain evaluation scope is null at seed-resolution, and the lifecycle is a net drag of approx -0.028 nats (t=-4.46,p=0.047 in the primary chain). An auxiliary alpha=0 inheritance counterfactual at n=3 seeds is sign-inconsistent at the headline metric and underpowered for either an equivalence or load-bearing conclusion (corrected from an earlier arithmetic-mean aggregator that erroneously cleared inheritance; see Appendix B.11). A base-perturbation probe directionally refutes a "genomic-context" reframe of the lifecycle role. A controllable synthetic sandbox locates a substrate-conditional regime boundary: evolutionary search on the routing channel is load-bearing only when adapters are pre-aligned to the task; in every other regime tested it underperforms, ties, or actively degrades the gradient solution.

Related papers

• The Filter Metric is Safety-Critical: Phantom Advantages in Group-Relative RL under Shaped Rewards [1.667501390377158]
  Group-relative policy optimization can discard nocontrast rollout groups through dynamic sampling.<n>We identify and quantify a metricpredicate mismatch under composite shaped rewards.
  arXiv  Detail & Related papers  (2026-09-12T10:36:55Z)
• SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models [53.86918766240095]
  Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation.<n>We propose textbfState Anomaly Neutralization (SANE), which applies adaptive $tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure.
  arXiv  Detail & Related papers  (2026-08-23T10:41:07Z)
• Feature Priming in Online Linear Regression: Sparse-Regret Lower Bounds and a Tight Univariate Rate [47.36630149538994]
  In high-dimensional online prediction, the best predictor may depend on only a few features, so regret should scale with sparsity rather than the ambient dimension.<n>Warmuth and Amid asked at COLT 2023 whether any of three such rules admits a competitive online regret guarantee.<n>Our analysis identifies a common obstruction: cheap nuisance causes the refit to underweight the truly predictive coordinate.
  arXiv  Detail & Related papers  (2026-08-18T09:32:03Z)
• Architecture-Dependent Causal Transfer of Activation States Across Large Language Models [51.56484100374058]
  Internal language activation states can be transferred causally between different large language models.<n>Injecting projected activations into a target model during generation produces a statistically significant, preregistered causal effect.<n>We conclude that end-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal.
  arXiv  Detail & Related papers  (2026-08-17T09:53:27Z)
• Structural Kolmogorov-Arnold Convolutions: Learnable Function on the Values or the Filter Shape as Parameter-Efficient Alternative to Per-Edge Convolutional KANs [76.06235645266621]
  Convolutional Kolmogorov--Arnold Networks (KANs) replace the fixed weights of a convolutional kernel with learnable univariate functions.<n>We argue that the learnable functions are better placed in the emphstructure of the convolution than on each edge.
  arXiv  Detail & Related papers  (2026-06-23T10:00:26Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• CRMA: A Spectrally-Bounded Backbone for Modular Continual Fine-Tuning of LLMs [0.0]
  We introduce CRMA, a residual adapter whose internal mixing matrix M is doubly-stochastic at every forward pass via Sinkhorn normalization.<n>On Mistral-7B across 5 domains 3 seeds, modular per-task LoRA mediates a backbone loss-relative drift from +42.96% +/- 5.5 (naive fine-tuning) to -0.17% +/- 0.17, with disjoint per-seed ranges, and improves prior-task holdout loss by 1.99% +/- 0.54 over a matched frozen-substrate baseline.
  arXiv  Detail & Related papers  (2026-05-29T21:50:23Z)
• Saturating Scaling Laws for Equational Discovery: A Phenomenology of Growth Dynamics in Three Toy Substrates with Two Real-World Replications [0.0]
  We investigate growth dynamics in deterministic equational discovery substrates.<n>Short-range substrate sizes fit a power-law N(t) proportional to tb.<n>A saturating power-law dN/dt = K Nk exp(-mu N) of which the pure power-law is the short-range approximation.
  arXiv  Detail & Related papers  (2026-05-14T21:37:29Z)
• EqOD: Symmetry-Informed Stability Selection for PDE Identification [3.1861308132183375]
  We introduce Equivariant Operator Discovery (EqOD), a fully automatic method combining two library reduction mechanisms.<n>When Galilean invariance is detected from trajectory data via a weak-form structural test, EqOD uses the symmetry-reduced library, eliminating terms that our Galilean exclusion result proves to be absent from the governing equation.<n>Across all 32 cells, EqOD outperforms PySINDy 2.0.0 in 23 of 32 cells, and all 5 PySINDy wins occur on reaction PDEs.
  arXiv  Detail & Related papers  (2026-05-12T04:48:38Z)
• Signal Reshaping for GRPO in Weak-Feedback Agentic Code Repair [36.31472731207028]
  We study signal reshaping for standard GRPO under such feedback.<n>Experiments show a clear end-to-end gain: full signal-reshaped GRPO improves strict compile-and-semantic accuracy.
  arXiv  Detail & Related papers  (2026-05-08T05:41:25Z)
• Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimes [4.738949927143789]
  We investigate whether linearly decodable failure signals in hidden states can be leveraged to correct those failures.<n>We find that decodable failure structure supports post-generation reliability estimation even when the fixed linear steering family cannot exploit it for correction.
  arXiv  Detail & Related papers  (2026-05-07T05:58:38Z)
• Revisiting Auxiliary Losses for Conditional Depth Routing: An Empirical Study [31.968379218484746]
  Gate decision must propagate through many layers before it influences the language modeling (LM) loss.<n> Auxiliary losses are commonly stacked to stabilise training, yet the interactions among them -- particularly between a predictive auxiliary and explicit score supervision -- have not been systematically compared under controlled conditions.<n>We trace this to an off-policy oracle label that assumes all subsequent layers execute full, whereas gated execution routes only a fraction through full.
  arXiv  Detail & Related papers  (2026-04-19T03:20:40Z)
• Auxiliary Finite-Difference Residual-Gradient Regularization for PINNs [0.0]
  We study a hybrid design in which the governing PDE residual remains automatic-differentiation based, while finite differences appear only in a weak auxiliary term.<n>The FD term regularizes the residual field without replacing the PDE residual itself.<n>Across seeds 0-5 and 100k epochs, the most reliable tested configuration is a fixed shell weight of 5e-4 under the Kourkoutas-beta regime.
  arXiv  Detail & Related papers  (2026-04-15T23:09:19Z)
• The Coordinate System Problem in Persistent Structural Memory for Neural Architectures [0.0]
  We introduce the Dual-View Pheromone Pathway Network (DPPN), an architecture that routes sparse attention through a persistent pheromone field over latent slot transitions.<n>We identify a core principle: persistent memory requires a stable coordinate system.<n>We show that neither contrastive updates, multi-source distillation, Hungarian alignment, nor semantic decomposition resolves the instability when embeddings are learned from scratch.
  arXiv  Detail & Related papers  (2026-03-24T06:51:59Z)
• Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels [83.99688944263843]
  Weight-De Low-Rank Adaptation (DoRA) extends LoRA by decoupling weight magnitude from direction.<n>At d_in = 8192 and rank r = 384, a single module's norm requires about 512 MB of transient working memory in bf16.<n>A factored norm decomposes the squared norm into base, cross, and Gram terms computable through O(d_out r + r2) intermediates, eliminating the dense product.
  arXiv  Detail & Related papers  (2026-03-23T17:57:24Z)
• Self-supervised Domain Adaptation for Visual 3D Pose Estimation of Nano-drone Racing Gates by Enforcing Geometric Consistency [43.497514957094275]
  We consider the task of visually estimating the relative pose of a drone racing gate in front of a nano-quadrotor, using a convolutional neural network pre-trained on simulated data to regress the gate's pose.<n>We propose an unsupervised domain adaptation (UDA) approach using only real image sequences collected by the drone flying an arbitrary trajectory in front of a gate.
  arXiv  Detail & Related papers  (2026-03-03T12:46:25Z)
• Quantile Advantage Estimation for Entropy-Safe Reasoning [44.192277495613695]
  Reinforcement Learning with Verifiable Rewards (RLVR) strengthens LLM reasoning, but training often oscillates between entropy collapse and entropy explosion<n>We trace both hazards to the mean baseline used in value-free RL, which improperly penalizes negative-advantage samples under reward outliers.<n>We propose Quantile Advantage Estimation (QAE), replacing the mean with a group-wise K-quantile baseline.
  arXiv  Detail & Related papers  (2025-09-26T17:37:52Z)
• Probabilistic Contrastive Learning for Domain Adaptation [42.33633916857581]
  Contrastive learning has shown impressive success in enhancing feature discriminability for various visual tasks in a self-supervised manner. Standard contrastive paradigm (features+$ell_2$ normalization) has limited benefits when applied in domain adaptation. We propose Probabilistic Contrastive Learning (PCL), which moves beyond the standard paradigm by removing $ell_2$ normalization and replacing the features with probabilities. PCL can guide the probability distribution towards a one-hot configuration, thus minimizing the discrepancy between features and class weights.
  arXiv  Detail & Related papers  (2021-11-11T02:08:07Z)
• Stratified Rule-Aware Network for Abstract Visual Reasoning [46.015682319351676]
  Raven's Progressive Matrices (RPM) test is typically used to examine the capability of abstract reasoning. Recent studies, taking advantage of Convolutional Neural Networks (CNNs), have achieved encouraging progress to accomplish the RPM test. We propose a Stratified Rule-Aware Network (SRAN) to generate the rule embeddings for two input sequences.
  arXiv  Detail & Related papers  (2020-02-17T08:44:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.