Omega-S: A Functional Resilience Index for LLM Fine-Tuning

• URL: http://arxiv.org/abs/2608.03887v1
• Date: Tue, 04 Aug 2026 16:22:30 GMT
• Title: Omega-S: A Functional Resilience Index for LLM Fine-Tuning
• Abstract summary: Fine-tuning a large language model degrades what it previously learned.<n>We present Omega-S, a drop-in penalty computed from the weight matrix alone.<n>It is three lines in an existing training loop and adds under 4% to the cost of a step.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Fine-tuning a large language model on new data degrades what it previously learned. We present Omega-S, a drop-in penalty computed from the weight matrix alone: it needs no previous-task data, no Fisher matrix and no stored copy of the old weights. It is three lines in an existing training loop and adds under 4% to the cost of a step. Retention. On Llama-3-8B with LoRA, fine-tuned from code to prose and measured by HumanEval over ten seeds, Omega-S retains more of the original capability than no regularisation on 9 of 10 seeds (0.173 -> 0.238 absolute pass@1; sign test one-sided p=0.011, Wilcoxon p=0.006), as a retention ratio, 62.9% -> 84.1%. It also beats tuned weight decay on 10 of 10 seeds (p=0.002) and tuned EWC on 8 of 10 (p=0.014), every arm re-measured in the same session. Mechanism, measured rather than asserted. Omega-S is topological by construction, its objective built from Tr(A^3), but we measured which of its four factors actually moves and three do not: their elasticity with respect to the weights is at or below 1e-4, against 9e-3 for the degree-variance term. As implemented, the composite reduces to a penalty on the variance of node degrees, which means row magnitude in square modules and directional alignment in non-square ones. We report this because a method whose name promises one thing and whose gradient does another should say so. We also enumerate the open design choices, including a contrast-preserving construction that does what it was designed to do and makes retention worse on all ten seeds. Repeating an identical configuration, same seed and same hardware, gives a standard deviation of 0.104 in retention ratio. We have not found this quantified for low-rank fine-tuning of language models, and it bounds every seed-paired comparison in this literature, ours included. Code, per-seed results and the full record of negative results are available.

Related papers

• Steering Under Compression: Dose-Response, Capability Cost, and Failure Asymmetry in Quantized LLMs [0.0]
  Inference-time steering enables behavioral control of large language models without parameter modification.<n>Post-training quantization reduces memory and compute costs for deployment.<n>We systematically study activation steering under weight-only quantization.
  arXiv  Detail & Related papers  (2026-09-06T08:41:42Z)
• Omega-N: Interpretable Structural Node Descriptors and Their Applicability Domain [0.0]
  A composite structural index summarises a network in one number.<n>A corollary in the theory paper for this index family stated that the global centrality should tie sharpened baselines rather than beat them.<n>We construct Omega-N by localizing each of the four factors.
  arXiv  Detail & Related papers  (2026-08-21T16:16:30Z)
• Feature Priming in Online Linear Regression: Sparse-Regret Lower Bounds and a Tight Univariate Rate [47.36630149538994]
  In high-dimensional online prediction, the best predictor may depend on only a few features, so regret should scale with sparsity rather than the ambient dimension.<n>Warmuth and Amid asked at COLT 2023 whether any of three such rules admits a competitive online regret guarantee.<n>Our analysis identifies a common obstruction: cheap nuisance causes the refit to underweight the truly predictive coordinate.
  arXiv  Detail & Related papers  (2026-08-18T09:32:03Z)
• When Single-Dataset Conclusions Fail: A 45-Task Study of Threshold Tuning and Resampling for Imbalanced Classification [0.0]
  Class-imbalance handling is routinely evaluated on a single benchmark dataset.<n>Under a leakage-free nested crossvalidation protocol, a plain Random Forest at the default 0.5 threshold attains F1 = 0.861 +/- 0.021.<n> Threshold-tuning benefit is non-monotonic in the imbalance ratio.
  arXiv  Detail & Related papers  (2026-08-17T05:58:33Z)
• Forgetful Attention: A Trainable Support-Vector Memory with Certified Selection and Exact Unlearning [0.16921396880325776]
  We introduce Support Vector Attention (SV-Attention), a memory whose weights are support coefficients of a one-class Sfp with fixed box parameter C.<n>Its active-set partition gives reserve tokens exactly zero weight, output weights; a incremental incremental solver token to recover the state produced by retraining without it under the same C.<n>We also demonstrate surgical eviction, exact editing, patient-record deletion, and a forgettable retrieval memory over real sentence embeddings.
  arXiv  Detail & Related papers  (2026-07-13T23:13:57Z)
• Privacy from Symmetry: Orthogonally Equivariant Transformers for LLM Inference [45.663066206650115]
  Split inference is often impractical, pushing inference on sensitive text to third-party providers.<n>We introduce ConjFormer, a transformer variant that is exactly $mathrmO(d)$-equivariant via a lightweight normalization change.
  arXiv  Detail & Related papers  (2026-06-15T09:31:24Z)
• Loss-Conditional PINNs for Parametric PDE Families [45.945826972787415]
  Physics-informed neural networks (PINNs) approximate solutions of ODEs and PDEs by minimising a weighted combination of residual, boundary, initial, and data losses.<n>We introduce LC-PINN, which adapts the lossconditional training of Dosovitskiy and Djolonga ( 2020) to the PDE-residual setting.<n>It stays fully physics-informed but amortises training over a parametric family.
  arXiv  Detail & Related papers  (2026-06-03T04:02:10Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• Measuring the Symmetry--Data Exchange Rate [0.2538209532048867]
  We show that a wrong-group control with identical orbit size and matched compute is worse than no constraint.<n>Relative exchange rate beta_diff = 1.28 is consistent in sign and order of magnitude with the theoretical bootstrap 1.0.<n>The methodological contributions -- the relative-rate estimator that cancels the shared-difficulty confound, the wrong-group control, and a pre-specified failure taxonomy -- transfer to any inductive bias whose strength can be parameterised.
  arXiv  Detail & Related papers  (2026-05-31T08:17:40Z)
• Decomposing Evolutionary Mixture-of-LoRA Architectures: The Routing Lever, the Lifecycle Penalty, and a Substrate-Conditional Boundary [3.151184728006369]
  We decompose an evolutionary mixtureofLoRA system into three factors -- a router rewrite, a perdomain leave-one-out evaluation scope, and a lifecycle of death plus alpha-blend inheritance plus SVD mutation plus slot reallocation.<n>A synthetic sandbox locates a substrate-conditional regime: evolutionary search on the routing channel is load-bearing only when are pre-aligned to the task; in every other regime tested it underperforms, ties, or actively degrades the solution.
  arXiv  Detail & Related papers  (2026-05-11T18:59:11Z)
• The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs [52.709361620508595]
  ListOPD brings a student to in-domain with an 8B-SFT baseline at one-fifth the parameters.<n>On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval test, and a small-clip cross-prediction--fall within their locked prediction windows--with the small-clip value matching the closed-form prediction below grid resolution.
  arXiv  Detail & Related papers  (2026-05-09T06:48:00Z)
• Statistically-Lossless Quantization of Large Language Models [41.38595517076645]
  This paper explores the middle ground of statistically-lossless compression through three complementary notions of losslessness for quantized LLMs.<n>First, task-lossless compression preserves zero-shot benchmark accuracy within natural sampling variance and remains achievable at aggressive bitwidths.<n>Second, we formalize the stricter notion of distribution-lossless compression, requiring the quantized model's next-token distribution to be practically indistinguishable from the original.<n>Third, we prove a gamma-squared variance law showing that symmetric quantization inflates noise variance by gamma squared relative to asymmetric quantization.
  arXiv  Detail & Related papers  (2026-05-04T09:46:47Z)
• Lost in State Space: Probing Frozen Mamba Representations [0.5156484100374058]
  Mamba's recurrent state h_t is a compressed summary of every token seen so far.<n>If we extract token-level outputs y_t at fixed patch boundaries, we obtain semantic sentence summaries for free.<n>We compare four strategies for extracting frozen sentence representations from a pretrained Mamba-130M backbone.
  arXiv  Detail & Related papers  (2026-04-30T21:35:08Z)
• ZenBrain: A Neuroscience-Inspired 7-Layer Memory Architecture for Autonomous AI Systems [51.56484100374058]
  On LongMemEval-500, ZenBrain matches a long-context oracle's binary-judge accuracy to within 4.5 pp.<n> ZenBrain is a 7-layer neuroscience-inspired memory architecture.
  arXiv  Detail & Related papers  (2026-04-26T20:39:19Z)
• Revisiting Auxiliary Losses for Conditional Depth Routing: An Empirical Study [31.968379218484746]
  Gate decision must propagate through many layers before it influences the language modeling (LM) loss.<n> Auxiliary losses are commonly stacked to stabilise training, yet the interactions among them -- particularly between a predictive auxiliary and explicit score supervision -- have not been systematically compared under controlled conditions.<n>We trace this to an off-policy oracle label that assumes all subsequent layers execute full, whereas gated execution routes only a fraction through full.
  arXiv  Detail & Related papers  (2026-04-19T03:20:40Z)
• Only relative ranks matter in weight-clustered large language models [0.0]
  Large language models (LLMs) contain billions of parameters, yet many exact values are not essential.<n>We show that what matters most is the relative rank of weights-whether one connection is stronger or weaker than another-rather than precise magnitudes.<n>For Llama 3.1-8B-Instruct and SmolLM2-135M, reducing each matrix to only 16-64 distinct values preserves strong accuracy without retraining.
  arXiv  Detail & Related papers  (2026-03-18T16:55:13Z)
• Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA [50.494504099850325]
  We introduce the Geodesic Hypothesis, positing that token sequences trace geodesics on a smooth semantic manifold and are therefore locally linear.<n>We show this constraint improves signal-to-noise ratio, and preserves diversity by preventing collisions during trajectory.<n>We demonstrate that geometric priors can surpass brute-force scaling.
  arXiv  Detail & Related papers  (2026-02-26T04:45:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.