LLM Attention Transplant for Transfer Learning of Tabular Data Across Disparate Domains

• URL: http://arxiv.org/abs/2511.06161v1
• Date: Sat, 08 Nov 2025 23:05:31 GMT
• Title: LLM Attention Transplant for Transfer Learning of Tabular Data Across Disparate Domains
• Authors: Ibna Kowsar, Kazi F. Akhter, Manar D. Samad,
• Abstract summary: We propose a lightweight transfer learning framework to transfer learning of mixed data types structured in tables.<n>Our experiments using ten pairs of source-target data sets and 12 baselines demonstrate the superiority of the proposed LLM-attention transplant for transfer learning.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Transfer learning of tabular data is non-trivial due to heterogeneity in the feature space across disparate domains. The limited success of traditional deep learning in tabular knowledge transfer can be advanced by leveraging large language models (LLMs). However, the efficacy of LLMs often stagnates for mixed data types structured in tables due to the limitations of text prompts and in-context learning. We propose a lightweight transfer learning framework that fine-tunes an LLM using source tabular data and transplants the LLM's selective $key$ and $value$ projection weights into a gated feature tokenized transformer (gFTT) built for tabular data. The gFTT model with cross-domain attention is fine-tuned using target tabular data for transfer learning, eliminating the need for shared features, LLM prompt engineering, and large-scale pretrained models. Our experiments using ten pairs of source-target data sets and 12 baselines demonstrate the superiority of the proposed LLM-attention transplant for transfer learning (LATTLE) method over traditional ML models, state-of-the-art deep tabular architectures, and transfer learning models trained on thousands to billions of tabular samples. The proposed attention transfer demonstrates an effective solution to learning relationships between data tables using an LLM in a low-resource learning environment. The source code for the proposed method is publicly available.

Related papers

• Distilling Transitional Pattern to Large Language Models for Multimodal Session-based Recommendation [67.84581846180458]
  Session-based recommendation (SBR) predicts the next item based on anonymous sessions.<n>Recent Multimodal SBR methods utilize simplistic pre-trained models for modality learning but have limitations in semantic richness.<n>We propose a multimodal LLM-enhanced framework TPAD, which extends a distillation paradigm to decouple and align transitional patterns for promoting MSBR.
  arXiv  Detail & Related papers  (2025-04-13T07:49:08Z)
• TableLoRA: Low-rank Adaptation on Table Structure Understanding for Large Language Models [57.005158277893194]
  TableLoRA is a module designed to improve LLMs' understanding of table structure during PEFT.<n>It incorporates special tokens for serializing tables with special token encoder and uses 2D LoRA to encode low-rank information on cell positions.
  arXiv  Detail & Related papers  (2025-03-06T12:50:14Z)
• LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularization [59.75242204923353]
  We introduce LLM-Lasso, a framework that leverages large language models (LLMs) to guide feature selection in Lasso regression.<n>LLMs generate penalty factors for each feature, which are converted into weights for the Lasso penalty using a simple, tunable model.<n>Features identified as more relevant by the LLM receive lower penalties, increasing their likelihood of being retained in the final model.
  arXiv  Detail & Related papers  (2025-02-15T02:55:22Z)
• Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models [15.603556124006479]
  We propose retrieval-augmented language models for scalable TabICL.<n>Our approach incorporates a customized retrieval module, combined with retrieval-guided instruction-tuning for LLMs.<n>This enables LLMs to effectively leverage larger datasets, achieving significantly improved performance across 69 widely recognized datasets.
  arXiv  Detail & Related papers  (2025-02-05T13:16:41Z)
• Transfer Learning of Tabular Data by Finetuning Large Language Models [0.0]
  This paper investigates the effectiveness of an application programming interface (API) and transfer learning of large language models (LLM)<n>LLM APIs respond to input text prompts with tokenized data and instructions, whereas transfer learning finetunes an LLM for a target classification task.<n>This paper proposes an end-to-end finetuning of LLM to demonstrate cross-data transfer learning on ten benchmark data sets.
  arXiv  Detail & Related papers  (2025-01-12T16:23:18Z)
• Tabular Transfer Learning via Prompting LLMs [52.96022335067357]
  We propose a novel framework, Prompt to Transfer (P2T), that utilizes unlabeled (or heterogeneous) source data with large language models (LLMs) P2T identifies a column feature in a source dataset that is strongly correlated with a target task feature to create examples relevant to the target task, thus creating pseudo-demonstrations for prompts.
  arXiv  Detail & Related papers  (2024-08-09T11:30:52Z)
• Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning [35.03338699349037]
  We propose a novel in-context learning framework, FeatLLM, which employs Large Language Models as feature engineers. FeatLLM generates high-quality rules, significantly (10% on average) outperforming alternatives such as TabLLM and STUNT.
  arXiv  Detail & Related papers  (2024-04-15T06:26:08Z)
• Small Models are LLM Knowledge Triggers on Medical Tabular Prediction [39.78560996984352]
  We propose SERSAL, a general self-prompting method by synergy learning with small models.<n>We show that SERSAL attains substantial improvement compared to linguistic prompting methods.
  arXiv  Detail & Related papers  (2024-03-03T17:35:52Z)
• Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes [57.62036621319563]
  We introduce CLLM, which leverages the prior knowledge of Large Language Models (LLMs) for data augmentation in the low-data regime. We demonstrate the superior performance of CLLM in the low-data regime compared to conventional generators.
  arXiv  Detail & Related papers  (2023-12-19T12:34:46Z)
• TAP4LLM: Table Provider on Sampling, Augmenting, and Packing Semi-structured Data for Large Language Model Reasoning [55.33939289989238]
  We propose TAP4LLM as a versatile pre-processor suite for leveraging large language models (LLMs) in table-based tasks effectively. It covers several distinct components: (1) table sampling to decompose large tables into manageable sub-tables based on query semantics, (2) table augmentation to enhance tables with additional knowledge from external sources or models, and (3) table packing & serialization to convert tables into various formats suitable for LLMs' understanding.
  arXiv  Detail & Related papers  (2023-12-14T15:37:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.