Mining Attribute Subspaces for Efficient Fine-tuning of 3D Foundation Models

• URL: http://arxiv.org/abs/2604.10095v1
• Date: Sat, 11 Apr 2026 08:33:13 GMT
• Title: Mining Attribute Subspaces for Efficient Fine-tuning of 3D Foundation Models
• Abstract summary: We introduce a robust approach that generates synthetic datasets with controlled variations, fine-tunes a LoRA adapter on each dataset, and extracts a LoRA sub-space associated with each type of variation.<n>We show that such a reduced LoRA subspace, despite being derived entirely from synthetic data, generalizes to real datasets.
• Score: 73.73784737382178
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the emergence of 3D foundation models, there is growing interest in fine-tuning them for downstream tasks, where LoRA is the dominant fine-tuning paradigm. As 3D datasets exhibit distinct variations in texture, geometry, camera motion, and lighting, there are interesting fundamental questions: 1) Are there LoRA subspaces associated with each type of variation? 2) Are these subspaces disentangled (i.e., orthogonal to each other)? 3) How do we compute them effectively? This paper provides answers to all these questions. We introduce a robust approach that generates synthetic datasets with controlled variations, fine-tunes a LoRA adapter on each dataset, and extracts a LoRA sub-space associated with each type of variation. We show that these subspaces are approximately disentangled. Integrating them leads to a reduced LoRA subspace that enables efficient LoRA fine-tuning with improved prediction accuracy for downstream tasks. In particular, we show that such a reduced LoRA subspace, despite being derived entirely from synthetic data, generalizes to real datasets. An ablation study validates the effectiveness of the choices in our approach.

Related papers

• Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropy [50.63908869296697]
  Merging multiple Low-Rank Adaptation (LoRA) modules is promising for constructing general-purpose systems.<n>We propose TARA-Merging, which aligns merging weights using a preference-weighted cross-entropy pseudo-loss.<n>Across eight vision and six NLI benchmarks, TARA-Merging consistently outperforms vanilla and LoRA-aware baselines.
  arXiv  Detail & Related papers  (2026-03-27T11:07:46Z)
• Latent Space Factorization in LoRA [10.994747174370099]
  Low-rank adaptation (LoRA) is a widely used method for parameter-efficient finetuning.<n>We propose Factorized Variational Autoencoder LoRA (FVAE-LoRA) to learn two distinct latent spaces.<n>Our novel Evidence Lower Bound formulation explicitly promotes factorization between the latent spaces, dedicating one latent space to task-salient features and the other to residual information.
  arXiv  Detail & Related papers  (2025-10-22T14:37:20Z)
• Not All LoRA Parameters Are Essential: Insights on Inference Necessity [36.65493658174926]
  We investigate the contribution of each LoRA layer to the model's ability to predict the ground truth.<n>We propose a simple yet effective method to enhance the performance of large language models fine-tuned with LoRA.
  arXiv  Detail & Related papers  (2025-03-30T08:33:04Z)
• BeamLoRA: Beam-Constraint Low-Rank Adaptation [51.52097743781401]
  Low-Rank Adaptation (LoRA) has been widely adopted as one of the most effective parameter-efficient fine-tuning methods.<n>We propose BeamLoRA, which conceptualizes each LoRA module as a beam where each rank naturally corresponds to a potential sub-solution.
  arXiv  Detail & Related papers  (2025-02-19T10:33:22Z)
• LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently [10.843508549704959]
  This paper explores how theory can guide and enhance practical algorithms, using Low-Rank Adaptation (LoRA) in large language models.<n>We rigorously prove that, under gradient descent, LoRA adapters align with specific singular subspaces of the one-step full fine-tuning gradient.<n>We propose a theory-driven algorithm, LoRA-One, where the linear convergence is built and incorporating preconditioners helps mitigate the effects of ill-conditioning.
  arXiv  Detail & Related papers  (2025-02-03T10:50:03Z)
• LoRA vs Full Fine-tuning: An Illusion of Equivalence [73.5303340531806]
  We study how Low-Rank Adaptation (LoRA) and full-finetuning change pre-trained models.<n>We find that LoRA and full fine-tuning yield weight matrices whose singular value decompositions exhibit very different structure.<n>We extend the finding that LoRA forgets less than full fine-tuning and find its forgetting is vastly localized to the intruder dimension.
  arXiv  Detail & Related papers  (2024-10-28T17:14:01Z)
• Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape [52.98187034726091]
  We introduce Flat-LoRA, which aims to identify a low-rank adaptation situated in a flat region of the full parameter space.<n>We show that Flat-LoRA improves both in-domain and out-of-domain generalization.
  arXiv  Detail & Related papers  (2024-09-22T11:24:10Z)
• NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models [27.757883818520217]
  Nested Low-Rank Adaptation (NoRA) is a novel approach to parameter-efficient fine-tuning. By freezing outer LoRA weights and using an inner LoRA design, NoRA enables precise task adaptation with a compact parameter space.
  arXiv  Detail & Related papers  (2024-08-18T12:18:56Z)
• Mixture-of-Subspaces in Low-Rank Adaptation [19.364393031148236]
  We introduce a subspace-inspired Low-Rank Adaptation (LoRA) method, which is computationally efficient, easy to implement, and readily applicable to large language, multimodal, and diffusion models.<n>To be more flexible, we jointly learn the mixer with the original LoRA weights, and term the method Mixture-of-Subspaces LoRA.<n>MoSLoRA consistently outperforms LoRA on tasks in different modalities, including commonsense reasoning, visual instruction tuning, and subject-driven text-to-image generation.
  arXiv  Detail & Related papers  (2024-06-16T14:19:49Z)
• LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks [72.88244322513039]
  LoRA employs lightweight modules to customize large language models (LLMs) for each downstream task or domain. We propose LoRA-Flow, which utilizes dynamic weights to adjust the impact of different LoRAs. Experiments across six generative tasks demonstrate that our method consistently outperforms baselines with task-level fusion weights.
  arXiv  Detail & Related papers  (2024-02-18T04:41:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.