AEGIS: Anchor-Enforced Gradient Isolation for Knowledge-Preserving Vision-Language-Action Fine-Tuning

• URL: http://arxiv.org/abs/2604.16067v1
• Date: Fri, 17 Apr 2026 13:49:57 GMT
• Title: AEGIS: Anchor-Enforced Gradient Isolation for Knowledge-Preserving Vision-Language-Action Fine-Tuning
• Abstract summary: Adapting pre-trained vision-language models (VLMs) for robotic control requires injecting high-magnitude continuous gradients from a flow-matching action expert into a backbone trained exclusively with cross-entropy.<n>This cross-modal gradient asymmetry causes rapid, severe erosion of the VLM's visual-question-answering (VQA) capability.<n>Industry-standard defences either sever the gradient pathway entirely via stop gradient, discarding the rich continuous supervision, or restrict parameter capacity through low-rank adapters (LoRA) that constrain the rank of updates but not their direction, and thus still overwrite the pre-trained manifold
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adapting pre-trained vision-language models (VLMs) for robotic control requires injecting high-magnitude continuous gradients from a flow-matching action expert into a backbone trained exclusively with cross-entropy. This cross-modal gradient asymmetry - the spectral dimensionality mismatch between low-rank MSE regression gradients and the high-dimensional semantic manifold sculpted by CE pre-training, causes rapid, severe erosion of the VLM's visual-question-answering (VQA) capability. Industry-standard defences either sever the gradient pathway entirely via stop gradient, discarding the rich continuous supervision, or restrict parameter capacity through low-rank adapters (LoRA) that constrain the rank of updates but not their direction, and thus still overwrite the pre-trained manifold. We introduce AEGIS (Anchor-Enforced Gradient Isolation System): a buffer-free, layer-wise orthogonal gradient projection framework that enables direct continuous MSE learning while preserving the pre-trained VQA manifold - without any co-training data or replay buffer. AEGIS pre-computes a static Gaussian reference anchor from masked VQA forward passes across all transformer layers, then at each training step constructs a Wasserstein-2 transport penalty that generates an anchor restoration gradient. A sequential dual-backward decomposes the task and anchor gradients; for each transformer layer, AEGIS applies a single Gram-Schmidt orthogonal projection that bends the task gradient away from the destructive direction while preserving its constructive content. The projection sheds less than 1% of gradient energy on average, yet eliminates the cumulative activation drift that drives severe forgetting.

Related papers

• Balancing Emotional Alignment and Semantic Consistency in Image Generation via Reinforcement Learning with Valence-Arousal Anchoring [55.57242007016976]
  Continuous emotion control in text-to-image generation requires a model to improve affective alignment without changing the objects, layout, or scene described by the prompt.<n>Existing supervised emotion-injection methods often optimize feature-space proxies and may therefore exhibit emotion-semantic drift.<n>We address this problem with a flow-matching image-generation framework that combines continuous valence-arousal conditioning, Group Relative Policy Optimization, and a neutral semantic anchor.
  arXiv  Detail & Related papers  (2026-09-11T13:24:35Z)
• Gradient Mirage: Trainable yet Label-Unidentifiable Gradients in Large Language Model Split Learning [21.543362368449156]
  gradient matching attacks (GMAs) in split learning rely on a critical yet underexplored assumption.<n>This consistency allows a curious server to recover private labels by searching for a sequence whose induced gradient explains the observation.<n>We propose Gradient Mirage, a defense that breaks this consistency without discarding the optimization utility of the backward signal.
  arXiv  Detail & Related papers  (2026-08-19T10:20:33Z)
• Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks [1.1516147824168732]
  Training binary neural networks (BNNs) from scratch is dominated by the straight-through estimator (STE)<n>We introduce StoMPP, which gradually replaces clipped weights and activations with their hard binary counterparts layer by layer from input to output.<n>As a standalone training rule, it provides a fully STE-free procedure that improves over vanilla STE with gains that grow with depth.
  arXiv  Detail & Related papers  (2026-06-26T06:37:57Z)
• CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry [34.36337082439934]
  Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers add little value.<n>We present two efficiency methods that exploit this asymmetry.<n> CascadeFormer tapers width with depth to match the uneven information flow across layers, achieving comparable perplexity to a uniform baseline at the same training budget while reducing latency by 8.6% and increasing throughput by 9.4%.
  arXiv  Detail & Related papers  (2026-06-25T02:25:00Z)
• SURGE: Surrogate Gradient Adaptation in Binary Neural Networks [41.424349870612716]
  We propose SURrogate GradiEnt Adaptation (SURGE), a learnable gradient compensation framework with theoretical grounding.<n> SURGE mitigates gradient mismatch through auxiliary backpropagation.<n> Experiments on image classification, object detection, and language understanding tasks demonstrate that SURGE performs best over state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-09T09:52:38Z)
• Enhancing Gradient Inversion Attacks in Federated Learning via Hierarchical Feature Optimization [56.95448807869383]
  Federated Learning (FL) has emerged as a compelling paradigm for privacy-preserving distributed machine learning.<n>Recent studies find that the gradients exchanged in the FL system are also vulnerable to privacy leakage.<n>We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD)
  arXiv  Detail & Related papers  (2026-04-01T14:32:15Z)
• Learn for Variation: Variationally Guided AAV Trajectory Learning in Differentiable Environments [21.594684328904403]
  Learn for Variation (L4V) is a gradient-informed trajectory learning framework.<n>It replaces high-variance scalar reward signals with dense and analytically grounded policy gradients.<n>L4V consistently outperforms representative baselines in mission completion time, average transmission rate, and training cost.
  arXiv  Detail & Related papers  (2026-03-19T12:57:42Z)
• Grad-ELLM: Gradient-based Explanations for Decoder-only LLMs [52.15785423211181]
  Grad-ELLM is a gradient-based attribution method for decoder-only transformer-based Large Language Models.<n>We introduce two faithfulneses metrics $$-Soft-NC and $$-Soft-NS, which provide fairer comparisons.<n>Experiment results show that Grad-ELLM consistently achieves superior faithfulness than other attribution methods.
  arXiv  Detail & Related papers  (2026-01-06T15:22:39Z)
• Geometric-Disentangelment Unlearning [106.99160454669902]
  gradient ascent on forget samples often harms retained knowledge.<n>We propose the Geometric-disment Unlearning (GU) that decomposes any candidate forget gradient update into tangential and normal components to retain space and executes only the normal component.<n>Our method is plug-and-play and can be attached to existing gradient-based unlearning procedures to mitigate side effects.
  arXiv  Detail & Related papers  (2025-11-21T09:58:25Z)
• Attention Saturation and Gradient Suppression at Inflection Layers: Diagnosing and Mitigating Bottlenecks in Transformer Adaptation [0.0]
  Pre-trained Transformers often exhibit over-confidence in source patterns and difficulty in forming new target-domain patterns during fine-tuning.<n>We formalize the mechanism of output saturation leading to gradient suppression through standard cross-entropy and softmax analysis.<n>We propose a diagnose-first, inject-light fine-tuning strategy: selectively inserting LoRA adapters at inflection layers to restore suppressed backward signals.
  arXiv  Detail & Related papers  (2025-11-02T04:32:41Z)
• GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling [39.3376897081385]
  We propose Gradient-Preserving Activation Scaling (GPAS), a technique that can be used in combination with existing approaches.<n>GPAS works by scaling down the intermediate activations while keeping their gradients unchanged.<n>Experiments across various model sizes from 71M to 1B show that GPAS achieves consistent performance gains.
  arXiv  Detail & Related papers  (2025-06-27T09:45:15Z)
• Mjolnir: Breaking the Shield of Perturbation-Protected Gradients via Adaptive Diffusion [13.764770382623812]
  We present the first attempt to break the shield of gradient perturbation protection in Federated Learning.<n>We introduce Mjolnir, a perturbation-resilient gradient leakage attack.<n>Mjolnir is capable of removing perturbations from gradients without requiring additional access to the original model structure or external data.
  arXiv  Detail & Related papers  (2024-07-07T07:06:49Z)
• Visual Prompt Tuning in Null Space for Continual Learning [51.96411454304625]
  Existing prompt-tuning methods have demonstrated impressive performances in continual learning (CL) This paper aims to learn each task by tuning the prompts in the direction orthogonal to the subspace spanned by previous tasks' features. In practice, an effective null-space-based approximation solution has been proposed to implement the prompt gradient projection.
  arXiv  Detail & Related papers  (2024-06-09T05:57:40Z)
• GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization [52.55628139825667]
  Federated Learning (FL) has emerged as a promising distributed machine learning framework to preserve clients' privacy. Recent studies find that an attacker can invert the shared gradients and recover sensitive data against an FL system by leveraging pre-trained generative adversarial networks (GAN) as prior knowledge. We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD), which disassembles the GAN model and searches the feature domains of the intermediate layers.
  arXiv  Detail & Related papers  (2023-08-09T04:34:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.