Sparse Fine-Tuning of Transformers for Generative Tasks

• URL: http://arxiv.org/abs/2507.10855v1
• Date: Mon, 14 Jul 2025 23:03:24 GMT
• Title: Sparse Fine-Tuning of Transformers for Generative Tasks
• Abstract summary: We introduce a fine-tuning framework inspired by sparse coding, where fine-tuned features are represented as a sparse combination of basic elements.<n>The feature dictionary atoms function as fundamental building blocks of the representation, and tuning atoms allows for seamless adaptation to downstream tasks.
• Score: 25.83571722912947
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large pre-trained transformers have revolutionized artificial intelligence across various domains, and fine-tuning remains the dominant approach for adapting these models to downstream tasks due to the cost of training from scratch. However, in existing fine-tuning methods, the updated representations are formed as a dense combination of modified parameters, making it challenging to interpret their contributions and understand how the model adapts to new tasks. In this work, we introduce a fine-tuning framework inspired by sparse coding, where fine-tuned features are represented as a sparse combination of basic elements, i.e., feature dictionary atoms. The feature dictionary atoms function as fundamental building blocks of the representation, and tuning atoms allows for seamless adaptation to downstream tasks. Sparse coefficients then serve as indicators of atom importance, identifying the contribution of each atom to the updated representation. Leveraging the atom selection capability of sparse coefficients, we first demonstrate that our method enhances image editing performance by improving text alignment through the removal of unimportant feature dictionary atoms. Additionally, we validate the effectiveness of our approach in the text-to-image concept customization task, where our method efficiently constructs the target concept using a sparse combination of feature dictionary atoms, outperforming various baseline fine-tuning methods.

Related papers

• Eigenfunction Extraction for Ordered Representation Learning [19.958531044442527]
  We propose a general framework to extract ordered and identifiable eigenfunctions, based on modular building blocks designed to satisfy key desiderata.<n>We validate our approach on synthetic kernels and demonstrate on real-world image datasets that the recovered eigenvalues act as effective importance scores for feature selection.
  arXiv  Detail & Related papers  (2025-10-28T17:37:12Z)
• Variational Task Vector Composition [53.476598858325985]
  We propose variational task vector composition, where composition coefficients are taken as latent variables and estimated in a Bayesian inference framework.<n>Motivated by the observation of structural redundancy in task vectors, we introduce a Spike-and-Slab prior that promotes sparsity.<n>We develop a gated sampling mechanism that constructs a controllable posterior by filtering the composition coefficients based on both uncertainty and importance.
  arXiv  Detail & Related papers  (2025-09-21T02:46:02Z)
• Structure-Learnable Adapter Fine-Tuning for Parameter-Efficient Large Language Models [5.019928514737434]
  The paper proposes an adapter-based fine-tuning method built on a structure-learnable mechanism.<n>It allows the model to adjust its structure flexibly in multi-task settings to match different task characteristics.<n>It achieves a better balance among accuracy, compression rate, and robustness to noise and perturbation.
  arXiv  Detail & Related papers  (2025-09-03T06:40:25Z)
• Optimal Invariant Bases for Atomistic Machine Learning [1.8849301976115886]
  Many machine learning models describe the local environment of atoms, but many are incomplete or functionally dependent.<n>We employ techniques from pattern recognition literature to existing atomistic representations to produce the smallest possible set that satisfies completeness.<n>We show that this yields a new message-passing network architecture that can recognize up to 5-body patterns in each neuron.
  arXiv  Detail & Related papers  (2025-03-30T16:52:29Z)
• "Principal Components" Enable A New Language of Images [79.45806370905775]
  We introduce a novel visual tokenization framework that embeds a provable PCA-like structure into the latent token space.<n>Our approach achieves state-of-the-art reconstruction performance and enables better interpretability to align with the human vision system.
  arXiv  Detail & Related papers  (2025-03-11T17:59:41Z)
• Atom dimension adaptation for infinite set dictionary learning [0.6554326244334866]
  We propose a method for adaptively adjusting the size of set-atoms in dictionary learning. The proposed algorithm not only decreases the representation error, but also improves anomaly detection.
  arXiv  Detail & Related papers  (2024-09-10T19:18:03Z)
• Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation [88.14365009076907]
  Iterative refinement is a useful paradigm for representation learning. We develop an implicit differentiation approach that improves the stability and tractability of training.
  arXiv  Detail & Related papers  (2022-07-02T10:00:35Z)
• Text Revision by On-the-Fly Representation Optimization [76.11035270753757]
  Current state-of-the-art methods formulate these tasks as sequence-to-sequence learning problems. We present an iterative in-place editing approach for text revision, which requires no parallel data. It achieves competitive and even better performance than state-of-the-art supervised methods on text simplification.
  arXiv  Detail & Related papers  (2022-04-15T07:38:08Z)
• Towards a Unified View of Parameter-Efficient Transfer Learning [108.94786930869473]
  Fine-tuning large pre-trained language models on downstream tasks has become the de-facto learning paradigm in NLP. Recent work has proposed a variety of parameter-efficient transfer learning methods that only fine-tune a small number of (extra) parameters to attain strong performance. We break down the design of state-of-the-art parameter-efficient transfer learning methods and present a unified framework that establishes connections between them.
  arXiv  Detail & Related papers  (2021-10-08T20:22:26Z)
• Adaptive Convolutions with Per-pixel Dynamic Filter Atom [24.691793951360914]
  We introduce scalable dynamic convolutions with per-pixel adapted filters. As plug-and-play replacements to convolutional layers, the introduced adaptive convolutions with per-pixel dynamic atoms enable explicit modeling of intra-image variance. We present experiments to show that, the proposed method delivers comparable or even better performance across tasks.
  arXiv  Detail & Related papers  (2021-08-17T22:04:10Z)
• Automated Concatenation of Embeddings for Structured Prediction [75.44925576268052]
  We propose Automated Concatenation of Embeddings (ACE) to automate the process of finding better concatenations of embeddings for structured prediction tasks. We follow strategies in reinforcement learning to optimize the parameters of the controller and compute the reward based on the accuracy of a task model.
  arXiv  Detail & Related papers  (2020-10-10T14:03:20Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)
• Supervised Learning for Non-Sequential Data: A Canonical Polyadic Decomposition Approach [85.12934750565971]
  Efficient modelling of feature interactions underpins supervised learning for non-sequential tasks. To alleviate this issue, it has been proposed to implicitly represent the model parameters as a tensor. For enhanced expressiveness, we generalize the framework to allow feature mapping to arbitrarily high-dimensional feature vectors.
  arXiv  Detail & Related papers  (2020-01-27T22:38:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.