TEMPER: Tensorized Efficient Manifold-constrained Parameterization for Expressive Residual Routing

• URL: http://arxiv.org/abs/2608.07851v1
• Date: Sat, 08 Aug 2026 01:45:27 GMT
• Title: TEMPER: Tensorized Efficient Manifold-constrained Parameterization for Expressive Residual Routing
• Abstract summary: Residual connections rely on a static residual pathway, and are essential for training deep neural networks.<n>We propose underlinetextbfTensorized underlinetextbfEfficient underlinetextbfManifold-constrained underlinetextbfParameterization for underlinetextbfExpressive Residual underlinetextbfRouting.
• Score: 7.909872430050534
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Residual connections rely on a static residual pathway, and are essential for training deep neural networks. Hyper-connections (HC) increase the expressivity of residual routing by incorporating multiple residual streams and learning dynamic information flow, while manifold-constrained (mHC) variants stabilize training through doubly stochastic residual mixing. However, a generator-level bottleneck remains in existing methods: they use dense, unstructured generators for pre-branch aggregation, residual mixing, and post-branch redistribution, which results in parameter count growing rapidly with the number of streams. To address this issue, we propose \underline{\textbf{T}}ensorized \underline{\textbf{E}}fficient \underline{\textbf{M}}anifold-constrained \underline{\textbf{P}}arameterization for \underline{\textbf{E}}xpressive Residual \underline{\textbf{R}}outing (\textbf{TEMPER}), which represents these generators as multi-way tensors over the input-stream, feature, and output-stream modes, and parameterizes them using tensor networks. Such a structured low-rank formulation is shown to preserve token-dependent manifold-constrained routing interface while substantially reducing parameter growth. It also promotes interpretability and intuition, as: i) tensor ranks control the dimensionality of the learned routing subspace, with full ranks recovering dense routing; while ii) the generator approximation errors bound differences in routing logits and, consequently, in the routed-block outputs. Comprehensive experiments show that TEMPER matches or outperforms existing methods across language modeling and commonsense reasoning tasks, while requiring substantially fewer additional parameters. At eight residual streams, TEMPER achieves the best CORE score while using about $84\%$ fewer additional parameters than mHC, thus showing a stronger performance-parameter efficiency trade-off.

Related papers

• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• Plug-and-Play Guidance for Discrete Diffusion Models via Gradient-Informed Logit Correction [7.637257579386074]
  Controllable generation with discrete diffusion models is often hindered by high computational overhead or the need for retraining.<n>We present underlinetextbfGradient-underlinetextbfInformed underlinetextbfCorrection (textbfGILC), a plug-and-play framework that efficiently estimates guidance signals.
  arXiv  Detail & Related papers  (2026-06-04T15:41:53Z)
• When Model Merging Breaks Routing: Training-Free Calibration for MoE [67.27839627141972]
  Hessian-Aware Router (HARC) is a training-free framework that leverages second-order curvature information to realign the merged router.<n> Experiments on mathematical reasoning and code generation tasks show that HARC effectively mitigates routing breakdown across diverse MoE merging baselines.
  arXiv  Detail & Related papers  (2026-06-02T09:33:33Z)
• Automatically Differentiable Nonlinear Tensor Networks (ADNTNs) for Exponential Compression of Deep Neural Networks [8.605564917115663]
  ADNTNs are a family of structured weight generators with compact core tensors trained end-to-end by reverse-mode automatic differentiation (AD)<n>The paper focuses on three architectures: Tree Networks (TTNs), augmented TTNs (aTTNs) with boundary disentanglers, and Multi-scale Entanglement Renormalisation (MERA)
  arXiv  Detail & Related papers  (2026-05-28T19:43:10Z)
• Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior [107.2098567818173]
  Latent Recurrent Transformer (LRT) is a lightweight augmentation of autoregressive transformers.<n>LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token.
  arXiv  Detail & Related papers  (2026-05-26T10:10:26Z)
• Discrete Stochastic Localization for Non-autoregressive Generation [18.211690435751446]
  We argue that the bottleneck is not continuity itself, but a representation in which denoising depends on timestep-indexed noise regimes.<n>We introduce emphDiscrete, a continuous-state framework with unit-sphere token embeddings.<n>One trained network then supports an entire family of per-token SNR paths, with endpoint masked-diffusion paths as a special case.
  arXiv  Detail & Related papers  (2026-05-13T00:12:24Z)
• Riemannian MeanFlow for One-Step Generation on Manifolds [54.09734511705173]
  Flow Matching enables simulation-free training of generative models.<n>MeanFlow can be extended to manifold-valued generation where velocities lie in location-dependent tangent spaces.<n>Experiments on spheres, tori, and SO(3) demonstrate competitive one-step sampling with improved quality-efficiency trade-offs and substantially reduced sampling cost.
  arXiv  Detail & Related papers  (2026-03-11T12:41:46Z)
• Variational Bayesian Flow Network for Graph Generation [54.94088904387278]
  We propose Variational Bayesian Flow Network (VBFN) for graph generation.<n>VBFN performs variational lifting to a tractable joint Gaussian variational belief family governed by structured precisions.<n>On synthetic and molecular graph datasets, VBFN improves fidelity and diversity, and surpasses baseline methods.
  arXiv  Detail & Related papers  (2026-01-30T03:59:38Z)
• Sparse by Rule: Probability-Based N:M Pruning for Spiking Neural Networks [66.61171793101872]
  Spiking neural networks (SNNs) promise energy-efficient intelligence via event-driven, sparse computation, but deeper architectures inflate parameters and computational cost, hindering their edge deployment.<n>Recent progress in SNN pruning helps alleviate this burden, yet existing efforts fall into only two families: emphunstructured pruning, which attains high sparsity but is difficult to accelerate on general hardware, and emphstructured pruning, which eases deployment but lack flexibility and often degrades accuracy at matched sparsity.<n>We introduce bfSpikeNM, the first SNN
  arXiv  Detail & Related papers  (2025-11-15T08:23:20Z)
• Kernel Regression of Multi-Way Data via Tensor Trains with Hadamard Overparametrization: The Dynamic Graph Flow Case [9.941965164307843]
  Kernel Regression via Trains with Hadamard overparametrization (KReTTaH) is a regression-based framework for interpretable multi-way data imputation.<n>KReTTaH consistently outperforms state-of-the-art alternatives.
  arXiv  Detail & Related papers  (2025-09-26T11:00:05Z)
• Boosting Pruned Networks with Linear Over-parameterization [8.796518772724955]
  Structured pruning compresses neural networks by reducing channels (filters) for fast inference and low footprint at run-time. To restore accuracy after pruning, fine-tuning is usually applied to pruned networks. We propose a novel method that first linearly over- parameterizes the compact layers in pruned networks to enlarge the number of fine-tuning parameters.
  arXiv  Detail & Related papers  (2022-04-25T05:30:26Z)
• Manifold Regularized Dynamic Network Pruning [102.24146031250034]
  This paper proposes a new paradigm that dynamically removes redundant filters by embedding the manifold information of all instances into the space of pruned networks. The effectiveness of the proposed method is verified on several benchmarks, which shows better performance in terms of both accuracy and computational cost.
  arXiv  Detail & Related papers  (2021-03-10T03:59:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.