Pin-Tuning: Parameter-Efficient In-Context Tuning for Few-Shot Molecular Property Prediction

• URL: http://arxiv.org/abs/2411.01158v1
• Date: Sat, 02 Nov 2024 07:06:30 GMT
• Title: Pin-Tuning: Parameter-Efficient In-Context Tuning for Few-Shot Molecular Property Prediction
• Authors: Liang Wang, Qiang Liu, Shaozhen Liu, Xin Sun, Shu Wu, Liang Wang,
• Abstract summary: We propose a parameter-efficient in-context tuning method, named Pin-Tuning. Specifically, we propose a lightweight adapter for pre-trained message passing layers (MP-Adapter) and Bayesian weight consolidation for pre-trained atom/bond embedding layers (Emb-BWC) When evaluated on public datasets, our method demonstrates superior tuning with fewer trainable parameters, improving few-shot predictive performance.
• Score: 25.436047251446023
• License:
• Abstract: Molecular property prediction (MPP) is integral to drug discovery and material science, but often faces the challenge of data scarcity in real-world scenarios. Addressing this, few-shot molecular property prediction (FSMPP) has been developed. Unlike other few-shot tasks, FSMPP typically employs a pre-trained molecular encoder and a context-aware classifier, benefiting from molecular pre-training and molecular context information. Despite these advancements, existing methods struggle with the ineffective fine-tuning of pre-trained encoders. We attribute this issue to the imbalance between the abundance of tunable parameters and the scarcity of labeled molecules, and the lack of contextual perceptiveness in the encoders. To overcome this hurdle, we propose a parameter-efficient in-context tuning method, named Pin-Tuning. Specifically, we propose a lightweight adapter for pre-trained message passing layers (MP-Adapter) and Bayesian weight consolidation for pre-trained atom/bond embedding layers (Emb-BWC), to achieve parameter-efficient tuning while preventing over-fitting and catastrophic forgetting. Additionally, we enhance the MP-Adapters with contextual perceptiveness. This innovation allows for in-context tuning of the pre-trained encoder, thereby improving its adaptability for specific FSMPP tasks. When evaluated on public datasets, our method demonstrates superior tuning with fewer trainable parameters, improving few-shot predictive performance.

Related papers

• Equivariant Masked Position Prediction for Efficient Molecular Representation [6.761418610103767]
  Graph neural networks (GNNs) have shown considerable promise in computational chemistry. We introduce a novel self-supervised approach termed Equivariant Masked Position Prediction (EMPP) EMPP formulates a nuanced position prediction task that is more well-defined and enhances the learning of quantum mechanical features.
  arXiv  Detail & Related papers  (2025-02-12T08:39:26Z)
• Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More [26.226145789963443]
  Mask-Enhanced Autoregressive Prediction (MEAP) is a training paradigm that seamlessly integrates Masked Language Modeling (MLM) into Next-Token Prediction (NTP) In intensive experiments, MEAP substantially outperforms NTP on key information retrieval and long-context reasoning tasks. Our analysis indicates that MEAP's effectiveness arises from its ability to promote more distinguishable attention scores by concentrating on a reduced set of non-masked tokens.
  arXiv  Detail & Related papers  (2025-02-11T11:49:03Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts. Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• Embedded Prompt Tuning: Towards Enhanced Calibration of Pretrained Models for Medical Images [18.094731760514264]
  We study the effectiveness of fine-tuning methods when adapting foundation models to medical image classification tasks. We propose the Embedded Prompt Tuning (EPT) method by embedding prompt tokens into the expanded channels. EPT outperforms several state-of-the-art finetuning methods by a significant margin on few-shot medical image classification tasks.
  arXiv  Detail & Related papers  (2024-07-01T06:35:53Z)
• Sparse is Enough in Fine-tuning Pre-trained Large Language Models [98.46493578509039]
  We propose a gradient-based sparse fine-tuning algorithm, named Sparse Increment Fine-Tuning (SIFT) We validate its effectiveness on a range of tasks including the GLUE Benchmark and Instruction-tuning.
  arXiv  Detail & Related papers  (2023-12-19T06:06:30Z)
• PACIA: Parameter-Efficient Adapter for Few-Shot Molecular Property Prediction [41.631010796783706]
  We propose PACIA, a parameter-efficient GNN adapter for few-shot MPP. We then adopt a hierarchical adaptation mechanism to adapt the encoder at task-level and the predictor at query-level.
  arXiv  Detail & Related papers  (2023-10-01T08:28:04Z)
• Approximated Prompt Tuning for Vision-Language Pre-trained Models [54.326232586461614]
  In vision-language pre-trained models, prompt tuning often requires a large number of learnable tokens to bridge the gap between the pre-training and downstream tasks. We propose a novel Approximated Prompt Tuning (APT) approach towards efficient VL transfer learning.
  arXiv  Detail & Related papers  (2023-06-27T05:43:47Z)
• Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Training [55.12082817901671]
  We propose a new self-supervised pre-training approach, named Masked and Permuted Vision Transformer (MaPeT) MaPeT employs autoregressive and permuted predictions to capture intra-patch dependencies. Our results demonstrate that MaPeT achieves competitive performance on ImageNet, compared to baselines and competitors under the same model setting.
  arXiv  Detail & Related papers  (2023-06-12T18:12:19Z)
• Parameter-Efficient Fine-Tuning without Introducing New Latency [7.631596468553607]
  We introduce a novel adapter technique that directly applies the adapter to pre-trained parameters instead of the hidden representation. Our proposed method attains a new state-of-the-art outcome in terms of both performance and storage efficiency, storing only 0.03% parameters of full fine-tuning.
  arXiv  Detail & Related papers  (2023-05-26T08:44:42Z)
• MixPHM: Redundancy-Aware Parameter-Efficient Tuning for Low-Resource Visual Question Answering [66.05768870785548]
  Finetuning pretrained Vision-Language Models (VLMs) has been a prevailing paradigm for achieving state-of-the-art performance in Visual Question Answering (VQA) Current parameter-efficient tuning methods dramatically reduce the number of tunable parameters, but there still exists a significant performance gap with full finetuning. We propose MixPHM, a redundancy-aware parameter-efficient tuning method that outperforms full finetuning in low-resource VQA.
  arXiv  Detail & Related papers  (2023-03-02T13:28:50Z)
• Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations [51.75960511842552]
  Fine-tuning of pretrained language models (PLMs) is prone to overfitting in the low resource scenarios. We present a novel method that operates on the hidden representations of a PLM to reduce overfitting.
  arXiv  Detail & Related papers  (2022-11-16T09:39:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.