Echo-LoRA: Parameter-Efficient Fine-Tuning via Cross-Layer Representation Injection

• URL: http://arxiv.org/abs/2605.08177v1
• Date: Tue, 05 May 2026 08:26:11 GMT
• Title: Echo-LoRA: Parameter-Efficient Fine-Tuning via Cross-Layer Representation Injection
• Abstract summary: We propose Echo-LoRA, a cross-layer representation injection method for parameter-efficient fine-tuning.<n>During training, Echo-LoRA collects boundary hidden states from deeper source layers, aggregates them into a sample-level echo representation, and uses lightweight projection networks to inject the resulting signal into shallow LoRA or DoRA modules.<n>On eight commonsense reasoning benchmarks, Echo-LoRA exceeds the reported LoRA baseline on average across LLaMA-7B, LLaMA2-7B, and LLaMA3-8B.
• Score: 13.686308852232857
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Parameter-efficient fine-tuning (PEFT) has become a practical route for adapting large language models to downstream tasks, with LoRA-style methods being particularly attractive because they are inexpensive to train and easy to deploy. Most LoRA variants, however, revise the update rule within the weight space of each layer and leave the intermediate representations formed by deeper layers largely unused. We propose Echo-LoRA, a cross-layer representation injection method for parameter-efficient fine-tuning. During training, Echo-LoRA collects boundary hidden states from deeper source layers, aggregates them into a sample-level echo representation, and uses lightweight projection and gating networks to inject the resulting signal into shallow LoRA or DoRA modules. Answer-only masking, masked distillation, and stochastic routing are used to keep this auxiliary path stable and to reduce the gap between training and inference. On eight commonsense reasoning benchmarks, Echo-LoRA exceeds the reported LoRA baselines by 5.7 percentage points on average across LLaMA-7B, LLaMA2-7B, and LLaMA3-8B. Under reproduced LoRA baselines in our unified implementation, the average gain is 3.0 points; when combined with DoRA, the gain is 2.7 points. The Echo path is discarded after training, so the deployed model keeps the original low-rank LoRA/DoRA form and adds neither inference-time parameters nor inference computation.

Related papers

• LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates [48.87937677659571]
  Low-rank adaptation (LoRA) is the standard way to finetune large models.<n>We introduce LoRA-TSD, an adapter that treats every LoRA step as a tangent to the fixed-rank matrix manifold.<n>We give the first global convergence guarantees for both LoRA-Pro and LoRA-TSD, with rates that drive the factor-gradient norms to zero.
  arXiv  Detail & Related papers  (2026-09-02T15:41:45Z)
• FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation [0.0]
  Low-rank adaptation (LoRA) assigns a uniform rank to every adapted weight matrix.<n>We compute the gradient variance of each LoRA-B matrix as a proxy for layer informativeness, and redistribute the rank budget proportionally.<n>The resulting adapter is a standard LoRA with a per-layer rank pattern.
  arXiv  Detail & Related papers  (2026-05-16T04:03:21Z)
• Strategic Over-Parameterization for Generalizable Low-Rank Adaptation [14.867641913391779]
  Adapting large language models to downstream tasks via full fine-tuning is increasingly impractical due to its computational and memory demands.<n>We introduce LoRA-Over, a framework grounded in a simple principle: enrich the optimization landscape during training, then collapse the enrichment at inference.<n>We evaluate LoRA-Over on language understanding (GLUE, T5-Base), dialogue (MT-Bench), arithmetic reasoning (GSM8K), and code generation (HumanEval) using LLaMA 2-7B and LLaMA 3.1-8B.
  arXiv  Detail & Related papers  (2026-05-15T12:26:17Z)
• GRIT -- Geometry-Aware PEFT with K-FACPreconditioning, Fisher-Guided Reprojection, andDynamic Rank Adaptation [4.748720471060117]
  GRIT is a curvature-aware LoRA procedure that preserves the LoRA parameterization.<n>It matches or surpasses LoRA and QLoRA while reducing trainable parameters by 46% on average.<n>GRIT yields lower drift and a better updates-vs-retention frontier than strong PEFT-optimizer baselines.
  arXiv  Detail & Related papers  (2026-01-01T06:31:54Z)
• Faster Than SVD, Smarter Than SGD: The OPLoRA Alternating Update [50.36542772932594]
  Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights.<n>There is still a gap between full training with low-rank projections (SVDLoRA) and LoRA fine-tuning, indicating that LoRA steps can be further improved.
  arXiv  Detail & Related papers  (2025-09-24T10:32:50Z)
• DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efficient Fine-Tuning [5.103108721904429]
  We introduce DropLoRA, a novel pruning-based approach that focuses on pruning the rank dimension.<n>By continuously adapting the learning subspace, DropLoRA significantly boosts performance without incurring additional training or infer- ence costs.
  arXiv  Detail & Related papers  (2025-08-24T12:45:36Z)
• A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models [22.457766373989365]
  Low-Rank Adapters (LoRAs) have been substantially adopted across various fields, including instruction tuning and domain adaptation.<n>To address the limited expressive capacity of LoRA, the Mixture-of-Expert (MoE) has been introduced for incorporating multiple LoRA adapters.<n>We propose a new training strategy for MoE-LoRA, to stabilize and boost its feature learning procedure by multi-space projections.
  arXiv  Detail & Related papers  (2025-02-20T05:58:53Z)
• BeamLoRA: Beam-Constraint Low-Rank Adaptation [51.52097743781401]
  Low-Rank Adaptation (LoRA) has been widely adopted as one of the most effective parameter-efficient fine-tuning methods.<n>We propose BeamLoRA, which conceptualizes each LoRA module as a beam where each rank naturally corresponds to a potential sub-solution.
  arXiv  Detail & Related papers  (2025-02-19T10:33:22Z)
• LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization [78.93425154518705]
  Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements.<n>This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization.
  arXiv  Detail & Related papers  (2024-10-27T22:57:12Z)
• MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning [9.91790333647256]
  Low-rank adaptation (LoRA) and its mixture-of-experts (MOE) variants are highly effective parameter-efficient fine-tuning (PEFT) methods. We propose Mixture of Low-Rank Adaptation (MiLoRA), a novel and efficient LoRA variant. MiLoRA differs from previous MOE-style LoRA methods by considering each LoRA module as an expert and employing a prompt-aware routing mechanism.
  arXiv  Detail & Related papers  (2024-10-23T17:04:40Z)
• ResLoRA: Identity Residual Mapping in Low-Rank Adaption [96.59370314485074]
  We propose ResLoRA, an improved framework of low-rank adaptation (LoRA) Our method can achieve better results in fewer training steps without any extra trainable parameters or inference cost compared to LoRA. The experiments on NLG, NLU, and text-to-image tasks demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2024-02-28T04:33:20Z)
• DoRA: Weight-Decomposed Low-Rank Adaptation [57.68678247436207]
  We introduce a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA. Aiming to resemble the learning capacity of FT from the findings, we propose Weight-Decomposed Low-Rank Adaptation (DoRA) DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning.
  arXiv  Detail & Related papers  (2024-02-14T17:59:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.