Dynamic Subspace Composition: Efficient Adaptation via Contractive Basis Expansion

• URL: http://arxiv.org/abs/2512.23448v1
• Date: Mon, 29 Dec 2025 13:11:01 GMT
• Title: Dynamic Subspace Composition: Efficient Adaptation via Contractive Basis Expansion
• Abstract summary: Mixture of Experts (MoE) models scale capacity but often suffer from representation collapse and gradient instability.<n>We propose Dynamic Subspace Composition (DSC), a framework that approximates context-dependent weights via a state-dependent, sparse expansion of a shared basis bank.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mixture of Experts (MoE) models scale capacity but often suffer from representation collapse and gradient instability. We propose Dynamic Subspace Composition (DSC), a framework that approximates context-dependent weights via a state-dependent, sparse expansion of a shared basis bank. Formally, DSC models the weight update as a residual trajectory within a Star- Shaped Domain, employing a Magnitude-Gated Simplex Interpolation to ensure continuity at the identity. Unlike standard Mixture-of-LoRAs, which incurs O(M rd) parameter complexity by retrieving independent rank-r matrices, DSC constructs a compositional rank-K approximation from decoupled unit-norm basis vectors. This reduces parameter complexity to O(M d) and memory traffic to O(Kd), while Frame-Theoretic regularization and spectral constraints provide rigorous worst-case bounds on the dynamic update. The code is available at https://github. com/VladimerKhasia/DSC

Related papers

• The data geometry of masking diffusion: Certified-optimal schedules via unmasking growth complexity [14.877292137182422]
  We study masking diffusion for discrete sampling and introduce a path-resolved measure of data geometry called the emphunmasking growth complexity (textsfUGCxspace)<n>Its local increments directly control Kullback--Leibler (KL) discretization error, yielding a unified analysis of Bernoulli-subset and fixed-cardinality unmasking schemes.
  arXiv  Detail & Related papers  (2026-08-13T17:40:17Z)
• Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation [54.74045834035952]
  Vision-language dataset distillation (VLDD) compresses a large image-text paired dataset into a small set of synthetic pairs.<n>We propose a rank-aware hyperbolic alignment (RAHA) that combines hierarchical geometry with explicit alignment-capacity control.<n>RAHA demonstrates competitive cross-modal retrieval and improved transfer indicators under fixed budgets.
  arXiv  Detail & Related papers  (2026-06-28T15:41:31Z)
• MambaADv2: Evolving Duality-enhanced State Space Model for Unsupervised Anomaly Detection [108.36437360254605]
  This paper proposes MambaADv2, a framework tailored for multi-class unsupervised anomaly detection.<n>MambaADv2 comprises a pre-trained encoder and a Mamba-inspired decoder, equipped with Duality-enhanced State Space (DSS) modules across multiple scales.<n>The structure of the Hybrid State Space (HSS) block is tailored by following the SSD-based Mamba lineage and incorporating Mamba3-style position-aware state-space modeling.
  arXiv  Detail & Related papers  (2026-06-22T10:14:06Z)
• Shift-Dependent Asymmetry: Orthogonal Inverse Low-Rank Adaptation for Federated Medical Segmentation [60.41721951405564]
  Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of segmentation foundation models for medical imaging.<n>Most federated LoRA methods adopt a uniform aggregation rule, which breaks under the encoder-decoder asymmetry in medical segmentation.<n>This mismatch entangles shared anatomy with site-specific biases and harms generalization.<n>We introduce a Subspace Orthogonality Regularizer that penalizes shared-local collinearity in the effective update space.
  arXiv  Detail & Related papers  (2026-06-07T15:39:28Z)
• LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models [56.999481798138625]
  LimiX-2M is a 2M- parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used benchmarks.<n>We present a unified tokenize-and-route framework for strong Tabular foundation models (TFMs)<n>Results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs.
  arXiv  Detail & Related papers  (2026-06-03T06:07:33Z)
• Empowering Heterogeneous Graph Foundation Models via Decoupled Relation Alignment [20.128308877213044]
  Decoupled relation Subspace Alignment (DRSA) is a novel, plug-and-play relation-driven framework.<n>It fundamentally shifts the paradigm by decoupling feature semantics from relation structures.<n>It can be seamlessly integrated as a universal preprocessing module.
  arXiv  Detail & Related papers  (2026-05-01T15:31:36Z)
• Inverting Self-Organizing Maps: A Unified Activation-Based Framework [39.146761527401424]
  We show that the activation pattern of a SOM can be inverted to recover the exact input under mild geometric conditions.<n>We introduce the Manifold-Aware Unified SOM Inversion and Control (MUSIC) update rule.<n>We validate the approach using synthetic Gaussian mixtures, the MNIST and the Faces in the Wild dataset.
  arXiv  Detail & Related papers  (2026-01-20T11:02:54Z)
• Structured Sparse Transition Matrices to Enable State Tracking in State-Space Models [68.31088463716269]
  We propose a structured sparse parametrization of transition matrices in state-space models (SSMs)<n>Our method, PD-SSM, parametrizes the transition matrix as the product of a column one-hot matrix ($P$) and a complex-valued diagonal matrix ($D$)<n>The model significantly outperforms a wide collection of modern SSM variants on various FSA state tracking tasks.
  arXiv  Detail & Related papers  (2025-09-26T12:46:30Z)
• DMSC: Dynamic Multi-Scale Coordination Framework for Time Series Forecasting [24.202509006348862]
  Time Series Forecasting (TSF) faces persistent challenges in modeling intricate temporal dependencies across different scales.<n>We propose a novel Dynamic Multi-Scale Coordination Framework (DMSC) with Multi-Scale Patch Decomposition block (EMPD), Triad Interaction Block (TIB) and Adaptive Scale Routing MoE block (ASR-MoE)<n>EMPD is designed as a built-in component to dynamically segment sequences into hierarchical patches with exponentially scaled granularities.<n>TIB then jointly models intra-patch, inter-patch, and cross-variable dependencies within each layer's decomposed representations.
  arXiv  Detail & Related papers  (2025-08-03T13:11:52Z)
• Sequential-Parallel Duality in Prefix Scannable Models [68.39855814099997]
  Recent developments have given rise to various models, such as Gated Linear Attention (GLA) and Mamba.<n>This raises a natural question: can we characterize the full class of neural sequence models that support near-constant-time parallel evaluation and linear-time, constant-space sequential inference?
  arXiv  Detail & Related papers  (2025-06-12T17:32:02Z)
• Exemplar-Free Continual Learning for State Space Models [32.73275711666184]
  State-Space Models (SSMs) excel at capturing long-range dependencies with structured recurrence.<n>Their evolving internal states pose challenges in adapting them under Continual Learning.<n>We propose Inf-SSM, a novel and simple geometry-aware regularization method.
  arXiv  Detail & Related papers  (2025-05-24T08:59:13Z)
• Recurrent Complex-Weighted Autoencoders for Unsupervised Object Discovery [62.43562856605473]
  We argue for the computational advantages of a recurrent architecture with complex-valued weights. We propose a fully convolutional autoencoder, SynCx, that performs iterative constraint satisfaction.
  arXiv  Detail & Related papers  (2024-05-27T15:47:03Z)
• Continuous Multidimensional Scaling [14.537835814280221]
  Multidimensional scaling (MDS) is the act of embedding proximity information about a set of $n$ objects in $d$-dimensional Euclidean space.<n>Here we are concerned with embedding dissimilarities by minimizing Kruskal's (1964) raw stress criterion.<n>Standard results from the theory of point-to-set maps can be used to establish that, if $n$ is fixed and a sequence of dissimilarity converges, then the limit of their embedded structures is the embedded structure of the limiting dissimilarity matrix.<n>We present such a reformulation, em continuous MDS.
  arXiv  Detail & Related papers  (2024-02-06T22:13:51Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• The Ordered Matrix Dirichlet for Modeling Ordinal Dynamics [54.96229007229786]
  We propose the Ordered Matrix Dirichlet (OMD) to map latent states to observed action types. Models built on the OMD recover interpretable latent states and show superior forecasting performance in few-shot settings.
  arXiv  Detail & Related papers  (2022-12-08T08:04:26Z)
• Learning Cross-view Geo-localization Embeddings via Dynamic Weighted Decorrelation Regularization [52.493240055559916]
  Cross-view geo-localization aims to spot images of the same location shot from two platforms, e.g., the drone platform and the satellite platform. Existing methods usually focus on optimizing the distance between one embedding with others in the feature space. In this paper, we argue that the low redundancy is also of importance, which motivates the model to mine more diverse patterns.
  arXiv  Detail & Related papers  (2022-11-10T02:13:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.