PACIA: Parameter-Efficient Adapter for Few-Shot Molecular Property Prediction

• URL: http://arxiv.org/abs/2310.00614v2
• Date: Wed, 8 May 2024 15:49:54 GMT
• Title: PACIA: Parameter-Efficient Adapter for Few-Shot Molecular Property Prediction
• Authors: Shiguang Wu, Yaqing Wang, Quanming Yao,
• Abstract summary: We propose PACIA, a parameter-efficient GNN adapter for few-shot MPP. We then adopt a hierarchical adaptation mechanism to adapt the encoder at task-level and the predictor at query-level.
• Score: 41.631010796783706
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Molecular property prediction (MPP) plays a crucial role in biomedical applications, but it often encounters challenges due to a scarcity of labeled data. Existing works commonly adopt gradient-based strategy to update a large amount of parameters for task-level adaptation. However, the increase of adaptive parameters can lead to overfitting and poor performance. Observing that graph neural network (GNN) performs well as both encoder and predictor, we propose PACIA, a parameter-efficient GNN adapter for few-shot MPP. We design a unified adapter to generate a few adaptive parameters to modulate the message passing process of GNN. We then adopt a hierarchical adaptation mechanism to adapt the encoder at task-level and the predictor at query-level by the unified GNN adapter. Extensive results show that PACIA obtains the state-of-the-art performance in few-shot MPP problems, and our proposed hierarchical adaptation mechanism is rational and effective.

Related papers

• Toward Relative Positional Encoding in Spiking Transformers [52.62008099390541]
  Spiking neural networks (SNNs) are bio-inspired networks that model how neurons in the brain communicate through discrete spikes. In this paper, we introduce an approximate method for relative positional encoding (RPE) in Spiking Transformers.
  arXiv  Detail & Related papers  (2025-01-28T06:42:37Z)
• OP-LoRA: The Blessing of Dimensionality [93.08208871549557]
  Low-rank adapters enable fine-tuning of large models with only a small number of parameters. They often pose optimization challenges, with poor convergence. We introduce an over- parameterized approach that accelerates training without increasing inference costs. We achieve improvements in vision-language tasks and especially notable increases in image generation.
  arXiv  Detail & Related papers  (2024-12-13T18:55:19Z)
• Pin-Tuning: Parameter-Efficient In-Context Tuning for Few-Shot Molecular Property Prediction [25.436047251446023]
  We propose a parameter-efficient in-context tuning method, named Pin-Tuning. Specifically, we propose a lightweight adapter for pre-trained message passing layers (MP-Adapter) and Bayesian weight consolidation for pre-trained atom/bond embedding layers (Emb-BWC) When evaluated on public datasets, our method demonstrates superior tuning with fewer trainable parameters, improving few-shot predictive performance.
  arXiv  Detail & Related papers  (2024-11-02T07:06:30Z)
• Generative Principal Component Regression via Variational Inference [2.4415762506639944]
  One approach to designing appropriate manipulations is to target key features of predictive models. We develop a novel objective based on supervised variational autoencoders (SVAEs) that enforces such information is represented in the latent space. We show in simulations that gPCR dramatically improves target selection in manipulation as compared to standard PCR and SVAEs.
  arXiv  Detail & Related papers  (2024-09-03T22:38:55Z)
• ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflections [59.839926875976225]
  We propose the ETHER transformation family, which performs Efficient fineTuning via HypErplane Reflections. In particular, we introduce ETHER and its relaxation ETHER+, which match or outperform existing PEFT methods with significantly fewer parameters.
  arXiv  Detail & Related papers  (2024-05-30T17:26:02Z)
• Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation [67.13876021157887]
  Dynamic Tuning (DyT) is a novel approach to improve both parameter and inference efficiency for ViT adaptation. DyT achieves superior performance compared to existing PEFT methods while evoking only 71% of their FLOPs on the VTAB-1K benchmark.
  arXiv  Detail & Related papers  (2024-03-18T14:05:52Z)
• CHAPTER: Exploiting Convolutional Neural Network Adapters for Self-supervised Speech Models [62.60723685118747]
  Self-supervised learning (SSL) is a powerful technique for learning representations from unlabeled data. We propose an efficient tuning method specifically designed for SSL speech model, by applying CNN adapters at the feature extractor. We empirically found that adding CNN to the feature extractor can help the adaptation on emotion and speaker tasks.
  arXiv  Detail & Related papers  (2022-12-01T08:50:12Z)
• Adaptive variational preparation of the Fermi-Hubbard eigenstates [0.0]
  We prepare highly accurate ground states of the Fermi-Hubbard model for small grids up to 6 sites (12bits) We show this adaptive method outperforms the non-adaptive counterpart in terms of fewer variational parameters, short gate depth, and scaling with the system size. We also demonstrate the application of adaptive variational methods by preparing excited states and Green functions using a proposed ADAPT-SSVQE algorithm.
  arXiv  Detail & Related papers  (2021-09-24T18:00:05Z)
• A Genetic Algorithm with Tree-structured Mutation for Hyperparameter Optimisation of Graph Neural Networks [8.02401104726362]
  Graph neural networks (GNNs) have gained increasing attention, as they possess excellent capability of processing graph-related problems. In practice, hyperparameter optimisation (HPO) is critical for GNNs to achieve satisfactory results. We propose a tree-structured mutation strategy for GA to alleviate this issue.
  arXiv  Detail & Related papers  (2021-02-24T00:31:52Z)
• A Study of Genetic Algorithms for Hyperparameter Optimization of Neural Networks in Machine Translation [0.0]
  We propose an automatic tuning method modeled after Darwin's Survival of the Fittest Theory via a Genetic Algorithm. Research results show that the proposed method, a GA, outperforms a random selection of hyper parameters.
  arXiv  Detail & Related papers  (2020-09-15T02:24:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.