Self-Supervised Pretraining of Molecular Graph Encoders with LeJEPA

• URL: http://arxiv.org/abs/2609.04261v1
• Date: Wed, 02 Sep 2026 12:08:23 GMT
• Title: Self-Supervised Pretraining of Molecular Graph Encoders with LeJEPA
• Abstract summary: Self-supervised pretraining has transformed language and vision, but its value for molecular graph neural networks remains contested.<n>We ask whether pretraining on a large unsupervised corpus improves molecular property prediction.<n>Pretraining improves learned representations but does not robustly improve finetuning.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised pretraining has transformed language and vision, but its value for molecular graph neural networks remains contested. We ask whether pretraining on a large unlabelled corpus improves molecular property prediction. We adapt LeJEPA, a predictor-free joint-embedding predictive architecture regularised by Sketched Isotropic Gaussian Regularisation (SIGReg), to molecular graphs, evaluating GPS and Chemprop-style D-MPNN encoders on the Wong et al. [1] antibiotic-activity dataset and ogbg-molhiv using a multi-seed, bootstrap-based protocol. Pretraining improves learned representations but does not robustly improve finetuning. A frozen probe on pretrained embeddings exceeds random initialisation on both tasks (ogbg-molhiv ROC-AUC 0.788 vs 0.665; +0.123), reaching the published self-supervised band, but this does not translate into finetuning gains. On the antibiotic scaffold split, a canonical partition is significant (delta AUPRC +0.041, p = 0.010), but the effect vanishes across five partitions (pooled +0.013, p = 0.095). Finetuning is null on the random split, ogbg-molhiv, and D-MPNN. The representational edge is nevertheless recoverable. Embeddings saturate at ~16-32 effective dimensions, whereas Morgan fingerprints improve to 1024 bits. At matched dimensionality, fingerprints lead validation (0.799 vs 0.782 at 128 dimensions) but trail shifted test scaffolds (0.759 vs 0.788). Truncating embeddings and combining them with a 1024-bit Morgan fingerprint raises ogbg-molhiv ROC-AUC from 0.805 to 0.832 (delta +0.027; 95% CI [+0.003, +0.054]; p = 0.014); an untrained encoder gains nothing (delta -0.003). Thus, pretraining supplies complementary information best realised through feature-level combination, while finetuning gains are weak and partition-dependent.

Related papers

• The Filter Metric is Safety-Critical: Phantom Advantages in Group-Relative RL under Shaped Rewards [1.667501390377158]
  Group-relative policy optimization can discard nocontrast rollout groups through dynamic sampling.<n>We identify and quantify a metricpredicate mismatch under composite shaped rewards.
  arXiv  Detail & Related papers  (2026-09-12T10:36:55Z)
• Denoised Variance-Based Pruning with Optimal Brain Bias Compensation [45.88028371034407]
  Variance-Based Pruning (VBP) introduced a promising paradigm by selecting neurons based on activation variance.<n>We introduce Denoised Variance-Based Pruning with Optimal Brain Bias Compensation (DVBP + OB$2$C)<n>It achieves state-of-the-art training-free performance; at 50% pruning, it retains over 90% of the original Top-1 accuracy.
  arXiv  Detail & Related papers  (2026-08-18T11:23:49Z)
• EGRL: Edge generation-guided relation-aware learning for RNA-protein interaction prediction [41.17218504310278]
  We propose Edge Generation-guided Relation-aware Learning (EGRL)<n>EGRL is a novel framework with several key components.<n> Comprehensive evaluations on four benchmark datasets demonstrate that EGRL achieves competitive overall performance.
  arXiv  Detail & Related papers  (2026-08-13T07:47:44Z)
• LeNEPA: No-Augmentation Next-Latent Prediction for Time-Series Representation Learning [59.34593956287438]
  We study how an SSL recipe behaves when its method-specific configuration is reused unchanged after the pretraining signal family changes.<n>We introduce Latent Euclidean Next-Embedding Prediction Architecture (LeNEPA)<n>LeNEPA replaces the stop-gradient/EMA stabilization used by vanilla NEPA with SIGReg-based isotropy regularization.
  arXiv  Detail & Related papers  (2026-07-01T13:56:21Z)
• Physics-Aware Auxiliary Losses Improve Out-of-Distribution Generalization of a GNN Synthesizability Filter [1.1986159241687198]
  We ask whether cheap, closed-form physical priors, used as auxiliary supervision on a graph neural network (GNN), improve OOD generalization.<n>We add two auxiliary losses to a GINE backbone: a topological complexity regression supervised by the Bertz index, and a strain-energy soft penalty supervised by MMFF94 force-field energy.<n>All three physics-aware variants give a small but statistically significant OOD improvement over the baseline.
  arXiv  Detail & Related papers  (2026-06-10T20:21:56Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Pretraining Objective Matters in Extreme Low-Data FGVC: A Backbone-Controlled Study [39.146761527401424]
  We compare four frozen ViT-B/16 encoders trained with supervised classification, contrastive learning (SigLIP2), masked reconstruction (MAE), and self-distillation (DINOv3)<n>Supervised and contrastive encoders provide the strongest linear separability (logistic AUC: 0.768 and 0.735; SVM AUC: 0.739 and 0.697)<n>We find that DINOv3 underperforms across probe families in this domain.
  arXiv  Detail & Related papers  (2026-05-15T04:14:16Z)
• HBEE: Human Behavioral Entropy Engine -- Pre-Registered Multi-Agent LLM Simulation of Peer-Suspicion-Based Detection Inversion [0.0]
  The paper makes a narrow but surprising claim: in a controlled environment where adaptive OPSEC is implementable as an LLM directive, peer-suspicion-cascade detection inverts.<n>We release the simulator, pre-registration document, frozen scenarios, raw telemetry, and analysis pipeline under an open-source license.
  arXiv  Detail & Related papers  (2026-05-08T09:19:21Z)
• Do Generative Metrics Predict YOLO Performance? An Evaluation Across Models, Augmentation Ratios, and Dataset Complexity [43.338311770275745]
  We present a controlled evaluation of synthetic augmentation for YOLOv11 across three single-class detection regimes.<n>We benchmark six GAN-, diffusion-, and hybrid-based generators over augmentation ratios from 10% to 150% of the real training split.<n>For each dataset-generator-augmentation configuration, we compute pre-training dataset metrics under a matched-size bootstrap protocol.
  arXiv  Detail & Related papers  (2026-02-20T03:02:36Z)
• Improved techniques for deterministic l2 robustness [63.34032156196848]
  Training convolutional neural networks (CNNs) with a strict 1-Lipschitz constraint under the $l_2$ norm is useful for adversarial robustness, interpretable gradients and stable training. We introduce a procedure to certify robustness of 1-Lipschitz CNNs by replacing the last linear layer with a 1-hidden layer. We significantly advance the state-of-the-art for standard and provable robust accuracies on CIFAR-10 and CIFAR-100.
  arXiv  Detail & Related papers  (2022-11-15T19:10:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.