IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning

• URL: http://arxiv.org/abs/2607.22251v1
• Date: Fri, 24 Jul 2026 12:34:46 GMT
• Title: IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning
• Abstract summary: Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning method for large language models.<n>We propose IFCLoRA, a topology-aware rank allocation method applied before fine-tuning.
• Score: 8.041472556779395
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning method for large language models, but its performance depends strongly on how a fixed rank budget is distributed across Transformer modules. Existing adaptive-rank methods usually rely on local gradient statistics collected during training, which introduces extra memory and computation and overlooks task-conditioned global information flow. We propose IFCLoRA, a topology-aware rank allocation method applied before fine-tuning. Using a small calibration set and a frozen pretrained model, IFCLoRA builds a sparse task-conditioned interaction graph whose nodes represent LoRA-compatible modules. It combines a global information-flow topology prior with local gradient sensitivity to compute Information-Flow Centrality scores, which estimate each module's adaptation importance under multi-hop propagation. Ranks are then assigned once under a global budget. Across multiple models, tasks, and low-rank settings, IFCLoRA consistently outperforms LoRA, AdaLoRA, and EVA under matched training configurations and total rank budgets, while retaining training costs comparable to standard LoRA. On mathematical reasoning with LLaMA 3 8B, IFCLoRA improves over LoRA by 1.36 percent at rank 4 and 1.82 percent at rank 8. Further analysis shows task-dependent, non-uniform rank profiles, indicating that global information-flow structure provides an informative and interpretable prior for low-budget parameter-efficient fine-tuning.

Related papers

• One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning [26.741445807383098]
  Low-rank adaptation (LoRA) has become the standard for parameter-efficient fine-tuning of large language models.<n>We show that rank-one components of a LoRA adapter update at highly uneven rates and low-velocity modules converge to concentrated singular spectra.<n>We propose an adaptive anisotropic learning-rate model that assigns each rank-one component its own effective learning rate.
  arXiv  Detail & Related papers  (2026-09-05T05:14:04Z)
• SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning [57.85676271833619]
  Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning.<n>As the rank increases, more principal singular directions are preserved, which generally improves the model's performance.<n>We propose SMoA, a textbfSpectrum textbfModulation textbfAdapter that enlarges the accessible family of spectrum-aware updates.
  arXiv  Detail & Related papers  (2026-05-20T13:19:28Z)
• FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation [0.0]
  Low-rank adaptation (LoRA) assigns a uniform rank to every adapted weight matrix.<n>We compute the gradient variance of each LoRA-B matrix as a proxy for layer informativeness, and redistribute the rank budget proportionally.<n>The resulting adapter is a standard LoRA with a per-layer rank pattern.
  arXiv  Detail & Related papers  (2026-05-16T04:03:21Z)
• MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning [45.41768510181293]
  MatryoshkaLoRA learns accurate hierarchical low-rank representations by inserting a fixed, carefully crafted diagonal matrix $P$ between the existing LoRA adapters.<n>Our results demonstrate that MatryoshkaLoRA learns more accurate hierarchical low-rank representations than prior rank-adaptive approaches.
  arXiv  Detail & Related papers  (2026-05-08T15:13:01Z)
• Post-Optimization Adaptive Rank Allocation for LoRA [10.796887002907456]
  Post-Optimization Adaptive Rank Allocation (PARA) is a data-free compression method for Low-Rank Adaptation (LoRA)<n>We propose PARA, a data-free compression method for LoRA that integrates seamlessly into existing fine-tuning pipelines.<n>We empirically demonstrate that PARA reduces parameter count by 75-90% while preserving the predictive performance of the original, uncompressed LoRA across vision and language benchmarks.
  arXiv  Detail & Related papers  (2026-04-30T12:40:11Z)
• Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning [48.66442009036754]
  Low-Rank Adaptation (LoRA) is the prevailing approach for efficient large language model fine-tuning.<n>In this work, we re-evaluate four representative LoRA variants alongside vanilla LoRA.<n>We find that different LoRA methods favor distinct learning rate ranges.
  arXiv  Detail & Related papers  (2026-02-04T19:36:20Z)
• BeamLoRA: Beam-Constraint Low-Rank Adaptation [51.52097743781401]
  Low-Rank Adaptation (LoRA) has been widely adopted as one of the most effective parameter-efficient fine-tuning methods.<n>We propose BeamLoRA, which conceptualizes each LoRA module as a beam where each rank naturally corresponds to a potential sub-solution.
  arXiv  Detail & Related papers  (2025-02-19T10:33:22Z)
• LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement [12.733972494875713]
  Foundation models (FMs) achieve strong performance across diverse tasks with task-specific fine-tuning.<n>Low-Rank Adaptation (LoRA) methods like Low-Rank Adaptation (LoRA) reduce this cost by introducing low-rank matrices for tuning fewer parameters.<n>LoRA-FAIR maintains computational and communication efficiency, yielding superior performance over state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-11-22T14:19:01Z)
• Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape [52.98187034726091]
  We introduce Flat-LoRA, which aims to identify a low-rank adaptation situated in a flat region of the full parameter space.<n>We show that Flat-LoRA improves both in-domain and out-of-domain generalization.
  arXiv  Detail & Related papers  (2024-09-22T11:24:10Z)
• LoRA-Pro: Are Low-Rank Adapters Properly Optimized? [121.0693322732454]
  Low-rank adaptation, also known as LoRA, has emerged as a prominent method for parameter-efficient fine-tuning of foundation models.<n>Despite its computational efficiency, LoRA still yields inferior performance compared to full fine-tuning.<n>We introduce LoRA-Pro, a method that enhances LoRA's performance by strategically adjusting the gradients of low-rank matrices.
  arXiv  Detail & Related papers  (2024-07-25T17:57:12Z)
• PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation [65.268245109828]
  We introduce PRILoRA, which linearly allocates a different rank for each layer, in an increasing manner, and performs pruning throughout the training process. We validate the effectiveness of PRILoRA through extensive experiments on eight GLUE benchmarks, setting a new state of the art.
  arXiv  Detail & Related papers  (2024-01-20T20:25:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.