Adaptive Capacity Allocation for Vision Language Action Fine-tuning

• URL: http://arxiv.org/abs/2603.07404v1
• Date: Sun, 08 Mar 2026 01:33:01 GMT
• Title: Adaptive Capacity Allocation for Vision Language Action Fine-tuning
• Abstract summary: Vision language action models (VLAs) are increasingly used for Physical AI, but deploying a pre-trained VLA model to unseen environments still requires adaptation.<n>We present LoRA-SP, a rank-adaptive fine-tuning method that replaces fixed-rank updates with input- and layer-wise capacity.<n>On four real-robot manipulation tasks collected on an unseen AgileX PiPER arm, LoRA-SP matches or exceeds full fine-tuning with far fewer trainable parameters.
• Score: 30.782665306687992
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision language action models (VLAs) are increasingly used for Physical AI, but deploying a pre-trained VLA model to unseen environments, embodiments, or tasks still requires adaptation. Parameter-efficient fine-tuning (PEFT), especially LoRA, is common for VLA policies, yet the exposed capacity knob, the rank, does not transfer uniformly: robotics transfer exhibits a higher and task-varying intrinsic rank than language fine-tuning. Small ranks suffice for LLMs (e.g., $r \in \{4, 8\}$), while spectral analyses indicate VLAs may require much larger ranks (e.g., $r \approx 128$) or near-full rank, a mismatch that worsens in multi-task settings. We present LoRA-SP (Select-Prune), a rank-adaptive fine-tuning method that replaces fixed-rank updates with input- and layer-wise capacity. LoRA-SP uses an SVD-style parameterization with a small router whose nonnegative scores act as singular values over a shared vector bank. The active set is chosen by an energy target on the cumulative squared scores $E(k) \ge η$, providing a direct link to approximation error via our spectral analysis. During training, $η$ concentrates energy on a few directions and teaches the router to rely on fewer vectors while preserving accuracy. This yields compact adapters that reduce cross-task interference and improve generalization. On four real-robot manipulation tasks collected on an unseen AgileX PiPER arm, across two VLA backbones ($π_0$ and SmolVLA), LoRA-SP matches or exceeds full fine-tuning with far fewer trainable parameters, and improves multi-task success by up to 31.6% over standard LoRA while remaining robust to rank choice.

Related papers

• Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning [2.749219725903528]
  Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially.<n>LoRA-based continual learning methods allocate a separate low-rank adapter per task, yet existing approaches either require task identity at inference or sum all adapters indiscriminately.<n>Recent gating-based solutions route inputs to the correct adapter but introduce trainable parameters that themselves need protection against forgetting.<n>Latent-LoRA is replay-free, requires no trainable routing component, and uses substantially fewer parameters per task.
  arXiv  Detail & Related papers  (2026-07-26T20:42:56Z)
• ReactVLA: Fast and Lightweight Reactive Robot Manipulation via Improved Mean Flow Action Generation [6.976481623762446]
  textttReactVLA is a lightweight and low-latency VLA framework for real-time robotic manipulation.<n>textttReactVLA consistently outperforms similarly sized VLA baselines.<n>It reduces real-world policy latency to below 38.6 ms, enabling fast reactive control on physical robot platforms.
  arXiv  Detail & Related papers  (2026-06-12T08:33:37Z)
• SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation [57.131411215593744]
  Fine-tuning vision-language-action (VLA) policies for long-horizon manipulation still relies heavily on behavior cloning.<n>We introduce RM, a multi-task stage-aware reward model that combines an action-primitive-based stage estimator with a multi-gate Mixture-of-Experts (MMoE) value head.<n>Building on RM, we further propose SPIRAL, an on-policy reward-guided framework that improves VLA policies from cheap autonomous rollouts.
  arXiv  Detail & Related papers  (2026-06-09T01:46:23Z)
• ALTO: Adaptive LoRA Tuning and Orchestration for Heterogeneous LoRA Training Workloads [9.260490240412308]
  Low-Rank Adaptation (LoRA) is now the dominant method for parameter-efficient fine-tuning of large language models.<n>LoRA performance is highly sensitive to configuration choices, leading to many concurrent LoRA jobs.<n>We present ALTO, a co-designed training system that accelerates LoRA hyperparameter tuning while enabling efficient cluster sharing.
  arXiv  Detail & Related papers  (2026-04-07T04:40:17Z)
• Spectral Surgery: Training-Free Refinement of LoRA via Gradient-Guided Singular Value Reweighting [19.886879714344076]
  Low-Rank Adaptation (LoRA) improves downstream performance by restricting task updates to a low-rank parameter subspace.<n>We find that trained LoRA updates often exhibit an inefficient spectrum: task effects concentrate in a small subset of singular directions.<n>We propose Spectral Surgery, a training-free refinement that decomposes a LoRA update with SVD.
  arXiv  Detail & Related papers  (2026-03-04T12:38:36Z)
• Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning [24.03926595342341]
  Low-Rank Adaptation (LoRA) is a tool for parameter-efficient finetuning of large models.<n>It is unclear how the optimal learning rate scales with adapter rank.<n>We introduce Maximal-Update Adaptation ($$A), a theoretical framework that characterizes how the "optimal" learning rate should scale.
  arXiv  Detail & Related papers  (2026-02-05T21:28:59Z)
• LoRA on the Go: Instance-level Dynamic LoRA Selection and Merging [9.68092924064735]
  Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for fine-tuning large language models.<n>LoGo is a training-free framework that dynamically selects and merges adapters at the instance level without any additional requirements.<n>LoGo outperforms training-based baselines on some tasks upto a margin of 3.6% while remaining competitive on other tasks.
  arXiv  Detail & Related papers  (2025-11-10T14:13:10Z)
• Faster Than SVD, Smarter Than SGD: The OPLoRA Alternating Update [50.36542772932594]
  Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights.<n>There is still a gap between full training with low-rank projections (SVDLoRA) and LoRA fine-tuning, indicating that LoRA steps can be further improved.
  arXiv  Detail & Related papers  (2025-09-24T10:32:50Z)
• Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights [75.83625828306839]
  textbfDrag-and-Drop LLMs (textitDnD) eliminates per-task training by mapping a handful of unlabeled task prompts directly to LoRA weight updates.<n>A lightweight text encoder distills each prompt batch into condition embeddings, which are then transformed by a cascaded hyper-convolutional decoder into the full set of LoRA matrices.
  arXiv  Detail & Related papers  (2025-06-19T15:38:21Z)
• Replay-Free Continual Low-Rank Adaptation with Dynamic Memory [62.85596937435928]
  We revisit continual learning, which enables pre-trained vision transformers (ViTs) to sequentially fine-tune on new downstream tasks over time.<n>Recent studies highlight a crossover between CL techniques and parameter-efficient fine-tuning.<n>We propose a novel PEFT-CL method called Dual Low-Rank Adaptation (DualLoRA)
  arXiv  Detail & Related papers  (2024-11-01T14:28:39Z)
• MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning [71.50432879573614]
  Low-rank adaptation (LoRA) is based on the idea that the adaptation process is intrinsically low-dimensional.<n>We present MELoRA, a mini-ensemble low-rank adapters that uses fewer trainable parameters while maintaining a higher rank.<n>Our experimental results show that, compared to LoRA, MELoRA achieves better performance with 8 times fewer trainable parameters on natural language understanding tasks and 36 times fewer trainable parameters on instruction following tasks.
  arXiv  Detail & Related papers  (2024-02-27T07:14:12Z)
• LoRA: Low-Rank Adaptation of Large Language Models [71.75808607987281]
  Low-Rank Adaptation, or LoRA, freezes the pre-trained model weights and injects trainable rank decomposition into each layer of the Transformer architecture. For GPT-3, LoRA can reduce the number of trainable parameters by 10,000 times and the computation hardware requirement by 3 times compared to full fine-tuning.
  arXiv  Detail & Related papers  (2021-06-17T17:37:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.