Molecules Meet Language: Confound-Aware Representation Learning and Chemical Property Steering in Transformer-VAE Latent Spaces

• URL: http://arxiv.org/abs/2605.06303v1
• Date: Thu, 07 May 2026 14:07:10 GMT
• Title: Molecules Meet Language: Confound-Aware Representation Learning and Chemical Property Steering in Transformer-VAE Latent Spaces
• Abstract summary: Molecular generative models often assume meaningful latent geometry, but apparent property predictability can reflect sequence-level shortcuts.<n>We study this issue in an unsupervised autoregressive Transformer-VAE trained on SELFIES.<n>We find chemically meaningful steering can emerge in entangled molecular latent spaces, but only when validated through decoded molecules.
• Score: 0.9332987715848714
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Molecular generative models often assume meaningful latent geometry, but apparent property predictability can reflect sequence-level shortcuts rather than chemical organization. We study this issue in an unsupervised autoregressive Transformer-VAE trained on SELFIES. After training, we freeze the model, fit linear probes to RDKit descriptors, and use the probe weights as candidate global steering directions. To separate chemical signal from SELFIES artifacts, we introduce a confound-aware evaluation based on residualization, confound-direction alignment analysis, and decoded-molecule traversal. This is necessary because SELFIES length, branch tokens, ring tokens, and token entropy are strongly encoded in the latent space. Under this confound-aware evaluation, we find robust monotonic steering for cLogP, FractionCSP3, HeavyAtomCount, TPSA, BertzCT, and HBA. Nonlinear probes further show that some properties admit stable global directions, while others are better described by local latent gradients. Overall, our results show that chemically meaningful steering can emerge in entangled molecular latent spaces, but only when validated through decoded molecules and controlled for representation-level confounds.

Related papers

• From Syntax to Semantics: Unveiling the Emergence of Chirality in SMILES Translation Models [1.481663141554356]
  We present Pan-CORE, a family of autoregressive Transformer-based encoder-decoder models for SMILES translation.<n>We use high-temporal-resolution checkpoint analysis to investigate how chiral information is learned during training.<n>Findings show that SMILES translation can serve as a useful experimental system for mechanistic analysis of semantic emergence in CLMs.
  arXiv  Detail & Related papers  (2026-05-11T03:53:49Z)
• SIGMA: Scalable Spectral Insights for LLM Collapse [51.863164847253366]
  We introduce SIGMA (Spectral Inequalities for Gram Matrix Analysis), a unified framework for model collapse.<n>By utilizing benchmarks that deriving and deterministic bounds on the matrix's spectrum, SIGMA provides a mathematically grounded metric to track the contraction of the representation space.<n>We demonstrate that SIGMA effectively captures the transition towards states, offering both theoretical insights into the mechanics of collapse.
  arXiv  Detail & Related papers  (2026-01-06T19:47:11Z)
• Aligned Manifold Property and Topology Point Clouds for Learning Molecular Properties [55.2480439325792]
  This work introduces AMPTCR, a molecular surface representation that combines local quantum-derived scalar fields and custom topological descriptors within an aligned point cloud format.<n>For molecular weight, results confirm that AMPTCR encodes physically meaningful data, with a validation R2 of 0.87.<n>In the bacterial inhibition task, AMPTCR enables both classification and direct regression of E. coli inhibition values.
  arXiv  Detail & Related papers  (2025-07-22T04:35:50Z)
• Pre-trained Molecular Language Models with Random Functional Group Masking [54.900360309677794]
  We propose a SMILES-based underlineem Molecular underlineem Language underlineem Model, which randomly masking SMILES subsequences corresponding to specific molecular atoms. This technique aims to compel the model to better infer molecular structures and properties, thus enhancing its predictive capabilities.
  arXiv  Detail & Related papers  (2024-11-03T01:56:15Z)
• Molecule Design by Latent Prompt Transformer [76.2112075557233]
  This work explores the challenging problem of molecule design by framing it as a conditional generative modeling task. We propose a novel generative model comprising three components: (1) a latent vector with a learnable prior distribution; (2) a molecule generation model based on a causal Transformer, which uses the latent vector as a prompt; and (3) a property prediction model that predicts a molecule's target properties and/or constraint values using the latent prompt.
  arXiv  Detail & Related papers  (2024-02-27T03:33:23Z)
• Learning Invariant Molecular Representation in Latent Discrete Space [52.13724532622099]
  We propose a new framework for learning molecular representations that exhibit invariance and robustness against distribution shifts. Our model achieves stronger generalization against state-of-the-art baselines in the presence of various distribution shifts.
  arXiv  Detail & Related papers  (2023-10-22T04:06:44Z)
• From Peptides to Nanostructures: A Euclidean Transformer for Fast and Stable Machine Learned Force Fields [5.013279299982324]
  We propose a transformer architecture called SO3krates that combines sparse equivariant representations with a self-attention mechanism. SO3krates achieves a unique combination of accuracy, stability, and speed that enables insightful analysis of quantum properties of matter on extended time and system size scales.
  arXiv  Detail & Related papers  (2023-09-21T09:22:05Z)
• CHA2: CHemistry Aware Convex Hull Autoencoder Towards Inverse Molecular Design [2.169755083801688]
  It is impossible to explore the entire search space comprehensively to exploit de novo structures with properties of interest. To address this challenge, reducing the intractable search space into a lower-dimensional latent volume helps examine molecular candidates more feasibly. We propose using a convex hall surrounding the top molecules in terms of high QEDs to ensnare a tight subspace in the latent representation as an efficient way to reveal novel molecules with high QEDs.
  arXiv  Detail & Related papers  (2023-02-21T21:05:31Z)
• Local manifold learning and its link to domain-based physics knowledge [53.15471241298841]
  In many reacting flow systems, the thermo-chemical state-space is assumed to evolve close to a low-dimensional manifold (LDM) We show that PCA applied in local clusters of data (local PCA) is capable of detecting the intrinsic parameterization of the thermo-chemical state-space.
  arXiv  Detail & Related papers  (2022-07-01T09:06:25Z)
• Characterizing the Latent Space of Molecular Deep Generative Models with Persistent Homology Metrics [21.95240820041655]
  Variational Autos (VAEs) are generative models in which encoder-decoder network pairs are trained to reconstruct training data distributions. We propose a method for measuring how well the latent space of deep generative models is able to encode structural and chemical features.
  arXiv  Detail & Related papers  (2020-10-18T13:33:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.