Efficient Training of Transformers for Molecule Property Prediction on Small-scale Datasets

• URL: http://arxiv.org/abs/2409.04909v1
• Date: Sat, 7 Sep 2024 21:07:12 GMT
• Title: Efficient Training of Transformers for Molecule Property Prediction on Small-scale Datasets
• Abstract summary: Blood-brain barrier (BBB) serves as a protective barrier that separates the brain from the circulatory system. Traditional experimental methods for measuring BBB permeability are challenging and impractical for large-scale screening. This paper proposes a GPS Transformer architecture augmented with Self Attention, designed to perform well in the low-data regime.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The blood-brain barrier (BBB) serves as a protective barrier that separates the brain from the circulatory system, regulating the passage of substances into the central nervous system. Assessing the BBB permeability of potential drugs is crucial for effective drug targeting. However, traditional experimental methods for measuring BBB permeability are challenging and impractical for large-scale screening. Consequently, there is a need to develop computational approaches to predict BBB permeability. This paper proposes a GPS Transformer architecture augmented with Self Attention, designed to perform well in the low-data regime. The proposed approach achieved a state-of-the-art performance on the BBB permeability prediction task using the BBBP dataset, surpassing existing models. With a ROC-AUC of 78.8%, the approach sets a state-of-the-art by 5.5%. We demonstrate that standard Self Attention coupled with GPS transformer performs better than other variants of attention coupled with GPS Transformer.

Related papers

• An Explainable Machine Learning Framework for Predicting Blood-Brain Barrier Permeability Using Molecular Descriptors [0.0]
  Blood-brain barrier (BBB) permeability is a critical determinant in the development of central nervous system therapeutics.<n>In this study, an explainable machine learning framework was developed to predict BBB permeability using molecular descriptors.
  arXiv  Detail & Related papers  (2026-09-09T10:42:56Z)
• Test-time Adaptation of Pelvic Bone Segmentation Models via Dynamic Reliability-Guided [62.10276009853225]
  Reliable pelvic bone segmentation from CT is essential for robot-assisted pelvic trauma surgery.<n>Existing methods show limited effectiveness for PBS.<n>We propose a novel closed-loop dynamic Reliability-Guided framework (ReGA) for PBS.
  arXiv  Detail & Related papers  (2026-08-01T08:13:20Z)
• Distributed Motion Planning with Safety Guarantees for Self-Reconfiguring Robotic Boats [76.98547580312116]
  This paper proposes a hybrid framework that combines distributed Model Predictive Control (MPC) with Control Barrier Functions (CBFs)<n>To ensure safety in real time, distributed CBF-based filters are applied to enforce inter-agent avoidance.
  arXiv  Detail & Related papers  (2026-07-22T16:37:05Z)
• SCALE:Scalable Conditional Atlas-Level Endpoint transport for virtual cell perturbation prediction [46.645872209565574]
  We present a specialized large-scale foundation model for virtual cell perturbation prediction.<n>First, we build a BioNeMo-based training and inference framework that substantially improves data throughput.<n>Second, we formulate perturbation prediction as conditional transport and implement it with a set-aware flow architecture.<n>Third, we evaluate the model on Tahoe-100M using a rigorous cell-level protocol centered on biologically meaningful metrics.
  arXiv  Detail & Related papers  (2026-03-18T05:51:08Z)
• Predictive Control with Indirect Adaptive Laws for Payload Transportation by Quadrupedal Robots [36.62645789321095]
  This paper develops a novel hierarchical planning and control framework for robust payload transportation by quadrupedal robots.<n>An indirect adaptive law estimates the unknown parameters of the reduced-order (template) locomotion model under varying payloads.<n>The robot successfully manages dynamic payloads with 73% of its mass on rough terrains.
  arXiv  Detail & Related papers  (2026-03-09T18:43:31Z)
• Dynamic Meta-Ensemble Framework for Efficient and Accurate Deep Learning in Plant Leaf Disease Detection on Resource-Constrained Edge Devices [0.0]
  We introduce a novel Dynamic Meta-Enemble Framework (DMEF) for high-accuracy plant disease diagnosis under resource constraints.<n>DMEF employs an adaptive weighting mechanism that dynamically combines the predictions of three lightweight convolutional neural networks.<n>Experiments on benchmark datasets for potato and maize diseases demonstrate state-of-the-art classification accuracies of 99.53% and 96.61%, respectively.
  arXiv  Detail & Related papers  (2026-01-24T03:57:49Z)
• Investigating Data Pruning for Pretraining Biological Foundation Models at Scale [47.09153330837959]
  We propose a post-hoc influence-guided data pruning framework tailored to biological domains.<n>Our framework consistently outperforms random selection baselines under an extreme pruning rate of over 99 percent.<n>These findings underscore the potential of influence-guided data pruning to substantially reduce the computational cost of BioFM pretraining.
  arXiv  Detail & Related papers  (2025-12-15T02:42:52Z)
• BiDense: Binarization for Dense Prediction [62.70804353158387]
  BiDense is a generalized binary neural network (BNN) designed for efficient and accurate dense prediction tasks. BiDense incorporates two key techniques: the Distribution-adaptive Binarizer (DAB) and the Channel-adaptive Full-precision Bypass (CFB)
  arXiv  Detail & Related papers  (2024-11-15T16:46:04Z)
• iFuzzyTL: Interpretable Fuzzy Transfer Learning for SSVEP BCI System [24.898026682692688]
  This study explores advanced classification techniques leveraging interpretable fuzzy transfer learning (iFuzzyTL) iFuzzyTL refines input signal processing and classification in a human-interpretable format by integrating fuzzy inference systems and attention mechanisms. The model's efficacy is demonstrated across three datasets.
  arXiv  Detail & Related papers  (2024-10-16T06:07:23Z)
• Applying Dimensionality Reduction as Precursor to LSTM-CNN Models for Classifying Imagery and Motor Signals in ECoG-Based BCIs [0.0]
  This research aims to elevate the field by optimizing motor imagery classification algorithms within Brain-Computer Interfaces (BCIs) We utilize unsupervised techniques for dimensionality reduction, namely Uniform Manifold Approximation and Projection (UMAP) and K-Nearest Neighbors (KNN) We also evaluate the necessity of employing supervised methods such as Long Short-Term Memory (LSTM) and Convolutional Neural Networks (CNNs) for classification tasks.
  arXiv  Detail & Related papers  (2023-11-22T16:34:06Z)
• EKGNet: A 10.96{\mu}W Fully Analog Neural Network for Intra-Patient Arrhythmia Classification [79.7946379395238]
  We present an integrated approach by combining analog computing and deep learning for electrocardiogram (ECG) arrhythmia classification. We propose EKGNet, a hardware-efficient and fully analog arrhythmia classification architecture that archives high accuracy with low power consumption.
  arXiv  Detail & Related papers  (2023-10-24T02:37:49Z)
• Non-Generative Energy Based Models [3.1447898427012473]
  Energy-based models (EBM) have become increasingly popular within computer vision. We propose a non-generative training approach, Non-Generative EBM (NG-EBM) We show that our NG-EBM training strategy retains many of the benefits of EBM in calibration, out-of-distribution detection, and adversarial resistance.
  arXiv  Detail & Related papers  (2023-04-03T18:47:37Z)
• A Computer Vision Enabled damage detection model with improved YOLOv5 based on Transformer Prediction Head [0.0]
  Current state-of-the-art deep learning (DL)-based damage detection models often lack superior feature extraction capability in complex and noisy environments. DenseSPH-YOLOv5 is a real-time DL-based high-performance damage detection model where DenseNet blocks have been integrated with the backbone. DenseSPH-YOLOv5 obtains a mean average precision (mAP) value of 85.25 %, F1-score of 81.18 %, and precision (P) value of 89.51 % outperforming current state-of-the-art models.
  arXiv  Detail & Related papers  (2023-03-07T22:53:36Z)
• Diffusion Denoising Process for Perceptron Bias in Out-of-distribution Detection [67.49587673594276]
  We introduce a new perceptron bias assumption that suggests discriminator models are more sensitive to certain features of the input, leading to the overconfidence problem. We demonstrate that the diffusion denoising process (DDP) of DMs serves as a novel form of asymmetric, which is well-suited to enhance the input and mitigate the overconfidence problem. Our experiments on CIFAR10, CIFAR100, and ImageNet show that our method outperforms SOTA approaches.
  arXiv  Detail & Related papers  (2022-11-21T08:45:08Z)
• Differentiable Agent-based Epidemiology [71.81552021144589]
  We introduce GradABM: a scalable, differentiable design for agent-based modeling that is amenable to gradient-based learning with automatic differentiation. GradABM can quickly simulate million-size populations in few seconds on commodity hardware, integrate with deep neural networks and ingest heterogeneous data sources.
  arXiv  Detail & Related papers  (2022-07-20T07:32:02Z)
• Neural Moving Horizon Estimation for Robust Flight Control [6.023276947115864]
  Estimating and reacting to external disturbances is crucial for robust flight control of quadrotors. We propose a neural moving horizon estimator (NeuroMHE) that can automatically tune the MHE parameters modeled by a neural network. NeuroMHE outperforms the state-of-the-art estimator with force estimation error reductions of up to 49.4%.
  arXiv  Detail & Related papers  (2022-06-21T13:43:24Z)
• Using Distance Correlation for Efficient Bayesian Optimization [0.0]
  We propose a BO scheme named BDC, which integrates BO with a statistical measure of association of two random variables called Distance Correlation.<n>BDC exploration balances and exploitation automatically, and requires no manual hyper parameter tuning.<n>We evaluate BDC on a range of benchmark tests and observe that it performs on per with popular BO methods.
  arXiv  Detail & Related papers  (2021-02-17T19:37:35Z)
• EEG-Inception: An Accurate and Robust End-to-End Neural Network for EEG-based Motor Imagery Classification [123.93460670568554]
  This paper proposes a novel convolutional neural network (CNN) architecture for accurate and robust EEG-based motor imagery (MI) classification. The proposed CNN model, namely EEG-Inception, is built on the backbone of the Inception-Time network. The proposed network is an end-to-end classification, as it takes the raw EEG signals as the input and does not require complex EEG signal-preprocessing.
  arXiv  Detail & Related papers  (2021-01-24T19:03:10Z)
• A Privacy-Preserving-Oriented DNN Pruning and Mobile Acceleration Framework [56.57225686288006]
  Weight pruning of deep neural networks (DNNs) has been proposed to satisfy the limited storage and computing capability of mobile edge devices. Previous pruning methods mainly focus on reducing the model size and/or improving performance without considering the privacy of user data. We propose a privacy-preserving-oriented pruning and mobile acceleration framework that does not require the private training dataset.
  arXiv  Detail & Related papers  (2020-03-13T23:52:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.