Superior Molecular Representations from Intermediate Encoder Layers

• URL: http://arxiv.org/abs/2506.06443v3
• Date: Wed, 15 Oct 2025 01:55:53 GMT
• Title: Superior Molecular Representations from Intermediate Encoder Layers
• Abstract summary: We analyze the information flow in five diverse molecular encoders and find that intermediate layers retain more general-purpose features.<n>We find that using frozen embeddings from optimal intermediate layers improves downstream performance by an average of 5.4%, up to 28.6%, compared to the final-layer.
• Score: 0.006860025762294234
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pretrained molecular encoders have become indispensable in computational chemistry for tasks such as property prediction and molecular generation. However, the standard practice of relying solely on final-layer embeddings for downstream tasks may discard valuable information. In this work, we first analyze the information flow in five diverse molecular encoders and find that intermediate layers retain more general-purpose features, whereas the final-layer specializes and compresses information. We then perform an empirical layer-wise evaluation across 22 property prediction tasks. We find that using frozen embeddings from optimal intermediate layers improves downstream performance by an average of 5.4%, up to 28.6%, compared to the final-layer. Furthermore, finetuning encoders truncated at intermediate depths achieves even greater average improvements of 8.5%, with increases as high as 40.8%, obtaining new state-of-the-art results on several benchmarks. These findings highlight the importance of exploring the full representational depth of molecular encoders to achieve substantial performance improvements and computational efficiency. The code will be made publicly available.

Related papers

• MDArena: Evaluating Coding Agents on Realistic Molecular Dynamics Workflows [20.54127577222597]
  We introduce MDArena, a benchmark of 50 containerized tasks drawn from active biomolecular simulation projects.<n>We evaluate six model/harness configurations spanning Codex and OpenCode.<n>Among the evaluated configurations, Codex GPT-5.5 at extra-high reasoning effort performs best, reaching 24/50 Strict successes@1 (48%), followed by Codex GPT-5.5 Medium with 21/50, and OpenCode Gemini Flash 3.5 with 20/50.
  arXiv  Detail & Related papers  (2026-07-31T13:48:07Z)
• Improving LLM Predictions via Inter-Layer Structural Encoders [3.3772986620114387]
  Inter-Layer Structurals (ILSE) is a powerful structural approach to learn one effective representation from the internal layer representations.<n>Cayley-Encoder is a mathematically grounded geometric encoder that leverages expander Cayley graphs for efficient inter-layer information propagation.<n>ILSE consistently outperforms baselines and existing approaches, achieving up to 44% improvement in accuracy and 25% in similarity metrics.
  arXiv  Detail & Related papers  (2026-03-24T00:37:44Z)
• Topological Feature Compression for Molecular Graph Neural Networks [0.0]
  We introduce a novel Graph Neural Network (GNN) architecture that combines compressed higher-order topological signals with standard molecular features.<n>Our approach captures global geometric information while preserving computational tractability and human-interpretable structure.
  arXiv  Detail & Related papers  (2025-08-11T09:45:47Z)
• Comparative Evaluation of Kolmogorov-Arnold Autoencoders and Orthogonal Autoencoders for Fault Detection with Varying Training Set Sizes [0.0]
  Kolmogorov-Arnold Networks (KANs) have emerged as a flexible and parameter-efficient alternative to conventional neural networks.<n>This study presents a comparative evaluation of KAN-based autoencoders (KAN-AEs) for unsupervised fault detection in chemical processes.
  arXiv  Detail & Related papers  (2025-08-04T19:41:12Z)
• Towards the Training of Deeper Predictive Coding Neural Networks [53.15874572081944]
  Predictive coding networks trained with equilibrium propagation are neural models that perform inference through an iterative energy process.<n>Previous studies have demonstrated their effectiveness in shallow architectures, but show significant performance degradation when depth exceeds five to seven layers.<n>We show that the reason behind this degradation is due to exponentially imbalanced errors between layers during weight updates, and predictions from the previous layer not being effective in guiding updates in deeper layers.
  arXiv  Detail & Related papers  (2025-06-30T12:44:47Z)
• Rethinking Decoder Design: Improving Biomarker Segmentation Using Depth-to-Space Restoration and Residual Linear Attention [2.0799865428691393]
  We propose an architecture that captures multi-scale local and global contextual information and a novel decoder design.<n>Our method achieves absolute performance gains of 2.76% on MoNuSeg, 3.12% on DSB, 2.87% on Electron Microscopy, and 4.03% on TNBC datasets.
  arXiv  Detail & Related papers  (2025-06-23T06:32:36Z)
• Enhancing Drug Discovery: Autoencoder-Based Latent Space Augmentation for Improved Molecular Solubility Prediction using LatMixSol [0.0]
  We propose LatMixSol, a novel latent space augmentation framework.<n>LatMixSol combines autoencoder-based feature compression with guided to enrich training data.<n>Our analysis confirms that cluster-guided latent space augmentation preserves chemical validity while expanding dataset diversity.
  arXiv  Detail & Related papers  (2025-05-30T20:54:57Z)
• COUNTDOWN: Contextually Sparse Activation Filtering Out Unnecessary Weights in Down Projection [3.647905567437244]
  sparse activation methods selectively deactivate non-essential parameters during inference.<n>We propose two methods: M-COUNTDOWN, leveraging indirect coefficients, and D-COUNTDOWN, utilizing direct coefficients of the linear combination.<n>Our specialized kernel implementations effectively realize these theoretical gains into substantial real-world acceleration.
  arXiv  Detail & Related papers  (2025-05-23T10:10:22Z)
• RocketPPA: Ultra-Fast LLM-Based PPA Estimator at Code-Level Abstraction [4.825037489691159]
  We introduce a novel framework that leverages a 21k dataset of thoroughly cleaned and synthesizable Verilog modules.<n>We fine-tune CodeLlama using LoRA-based parameter-efficient methods, framing the task as a regression problem to accurately predict PPA metrics from Verilog code.
  arXiv  Detail & Related papers  (2025-03-27T20:35:09Z)
• GAQAT: gradient-adaptive quantization-aware training for domain generalization [54.31450550793485]
  We propose a novel Gradient-Adaptive Quantization-Aware Training (GAQAT) framework for DG.<n>Our approach begins by identifying the scale-gradient conflict problem in low-precision quantization.<n>Extensive experiments validate the effectiveness of the proposed GAQAT framework.
  arXiv  Detail & Related papers  (2024-12-07T06:07:21Z)
• Beyond adaptive gradient: Fast-Controlled Minibatch Algorithm for large-scale optimization [1.6749379740049926]
  We introduce F-CMA, a Fast-Controlled Mini-batch Algorithm with a random reshuffling method featuring a sufficient decrease condition and a line-search procedure to ensure loss reduction per epoch.<n>Tests show significant improvements, including a decrease in the overall training time by 68%, an increase in per-epoch efficiency by up to 20%, and in model accuracy by up to 5%.
  arXiv  Detail & Related papers  (2024-11-24T11:46:47Z)
• A Fresh Take on Stale Embeddings: Improving Dense Retriever Training with Corrector Networks [81.2624272756733]
  In dense retrieval, deep encoders provide embeddings for both inputs and targets. We train a small parametric corrector network that adjusts stale cached target embeddings. Our approach matches state-of-the-art results even when no target embedding updates are made during training.
  arXiv  Detail & Related papers  (2024-09-03T13:29:13Z)
• PCP-MAE: Learning to Predict Centers for Point Masked Autoencoders [57.31790812209751]
  We show that when directly feeding the centers of masked patches to the decoder without information from the encoder, it still reconstructs well. We propose a simple yet effective method, i.e., learning to Predict Centers for Point Masked AutoEncoders (PCP-MAE) Our method is of high pre-training efficiency compared to other alternatives and achieves great improvement over Point-MAE.
  arXiv  Detail & Related papers  (2024-08-16T13:53:53Z)
• AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs [61.13296177652599]
  We show that data mixtures that perform well at smaller scales may not retain their advantage at larger scales.<n>We propose AutoScale, a two-stage, scale-aware data composition framework.
  arXiv  Detail & Related papers  (2024-07-29T17:06:30Z)
• LayerMatch: Do Pseudo-labels Benefit All Layers? [77.59625180366115]
  Semi-supervised learning offers a promising solution to mitigate the dependency of labeled data. We develop two layer-specific pseudo-label strategies, termed Grad-ReLU and Avg-Clustering. Our approach consistently demonstrates exceptional performance on standard semi-supervised learning benchmarks.
  arXiv  Detail & Related papers  (2024-06-20T11:25:50Z)
• Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation [67.85309547416155]
  A powerful architecture for universal segmentation relies on transformers that encode multi-scale image features and decode object queries into mask predictions.<n>With efficiency being a high priority for scaling such models, we observed that the state-of-the-art method Mask2Former uses 50% of its compute only on the transformer encoder.<n>This is due to the retention of a full-length token-level representation of all backbone feature scales at each encoder layer.
  arXiv  Detail & Related papers  (2024-04-23T01:34:20Z)
• Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE [62.13435256279566]
  Large Language Models (LLMs) have achieved remarkable performance across a wide variety of natural language tasks. However, their large size makes their inference slow and computationally expensive. We show that it enables these layers to acquire 'good' generation ability without affecting the generation ability of the final layer.
  arXiv  Detail & Related papers  (2023-10-28T04:07:58Z)
• Bi-level Contrastive Learning for Knowledge-Enhanced Molecule Representations [68.32093648671496]
  We introduce GODE, which accounts for the dual-level structure inherent in molecules.<n> Molecules possess an intrinsic graph structure and simultaneously function as nodes within a broader molecular knowledge graph.<n>By pre-training two GNNs on different graph structures, GODE effectively fuses molecular structures with their corresponding knowledge graph substructures.
  arXiv  Detail & Related papers  (2023-06-02T15:49:45Z)
• An Exploration of Encoder-Decoder Approaches to Multi-Label Classification for Legal and Biomedical Text [20.100081284294973]
  We compare four methods for multi-label classification, two based on an encoder only, and two based on an encoder-decoder. Our results show that encoder-decoder methods outperform encoder-only methods, with a growing advantage on more complex datasets.
  arXiv  Detail & Related papers  (2023-05-09T17:13:53Z)
• Towards Efficient Fine-tuning of Pre-trained Code Models: An Experimental Study and Beyond [52.656743602538825]
  Fine-tuning pre-trained code models incurs a large computational cost. We conduct an experimental study to explore what happens to layer-wise pre-trained representations and their encoded code knowledge during fine-tuning. We propose Telly to efficiently fine-tune pre-trained code models via layer freezing.
  arXiv  Detail & Related papers  (2023-04-11T13:34:13Z)
• Scalable training of graph convolutional neural networks for fast and accurate predictions of HOMO-LUMO gap in molecules [1.8947048356389908]
  This work focuses on building GCNN models on HPC systems to predict material properties of millions of molecules. We use HydraGNN, our in-house library for large-scale GCNN training, leveraging distributed data parallelism in PyTorch. We perform parallel training on two open-source large-scale graph datasets to build a GCNN predictor for an important quantum property known as the HOMO-LUMO gap.
  arXiv  Detail & Related papers  (2022-07-22T20:54:22Z)
• Transformer-based Context Condensation for Boosting Feature Pyramids in Object Detection [77.50110439560152]
  Current object detectors typically have a feature pyramid (FP) module for multi-level feature fusion (MFF) We propose a novel and efficient context modeling mechanism that can help existing FPs deliver better MFF results. In particular, we introduce a novel insight that comprehensive contexts can be decomposed and condensed into two types of representations for higher efficiency.
  arXiv  Detail & Related papers  (2022-07-14T01:45:03Z)
• Semantic Perturbations with Normalizing Flows for Improved Generalization [62.998818375912506]
  We show that perturbations in the latent space can be used to define fully unsupervised data augmentations. We find that our latent adversarial perturbations adaptive to the classifier throughout its training are most effective.
  arXiv  Detail & Related papers  (2021-08-18T03:20:00Z)
• Generate Novel Molecules With Target Properties Using Conditional Generative Models [0.0]
  We present a novel neural network for generating small molecules similar to the ones in the training set. Our network outperforms previous methods using Molecular weight, LogP and Quantitative Estimation of Drug-likeness as the evaluation metrics.
  arXiv  Detail & Related papers  (2020-09-15T18:59:26Z)
• SASL: Saliency-Adaptive Sparsity Learning for Neural Network Acceleration [20.92912642901645]
  We propose a Saliency-Adaptive Sparsity Learning (SASL) approach for further optimization. Our method can reduce 49.7% FLOPs of ResNet-50 with very negligible 0.39% top-1 and 0.05% top-5 accuracy degradation.
  arXiv  Detail & Related papers  (2020-03-12T16:49:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.