DISTAL: Distillation and Self-Supervised Pretraining for Structure-Agnostic Materials Property Prediction

• URL: http://arxiv.org/abs/2609.00059v1
• Date: Sun, 30 Aug 2026 14:00:55 GMT
• Title: DISTAL: Distillation and Self-Supervised Pretraining for Structure-Agnostic Materials Property Prediction
• Abstract summary: DISTAL is a dual-prior framework for structure-agnostic materials property prediction.<n>It combines self-supervised compositional pretraining with structure-aware knowledge distillation.<n> DISTAL achieves the strongest overall performance among all evaluated feature combinations.
• Score: 20.20663425813761
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Materials property prediction remains difficult in low-data settings, where many target properties are supported by only a limited number of labeled samples. Models with the strongest predictive accuracy often depend on crystal structures, which restricts their use in early-stage screening when structural information is limited or unavailable. To address this challenge, we propose DISTAL, a dual-prior framework for structure-agnostic materials property prediction that combines self-supervised compositional pretraining with structure-aware knowledge distillation. DISTAL first learns transferable compositional representations from a large virtual composition space using 145 composition-derived descriptors. It then distills structural knowledge from a pretrained ALIGNN teacher into a composition-conditioned student. This setting allows structural priors to be used during training without requiring structural inputs at inference. By integrating explicit compositional descriptors, pretrained latent features, and distilled structural features within a unified prediction pipeline, DISTAL captures complementary signals that are difficult to recover from any single representation alone. Across 39 benchmark tasks, the best-performing multimodal configuration combines all three signals, and improves over the reference benchmark on 37 tasks. DISTAL achieves the strongest overall performance among all evaluated feature combinations. These results indicate that compositional pretraining and structural distillation provide complementary priors and offer a practical route to robust composition-only prediction in small-data materials informatics. The source code and the pre-trained models are anonymously available at: https://osf.io/eq96d/overview?view_only=451617f42f7849e08750bd1852b48980 and will be released at the official link after acceptance.

Related papers

• From Synthetic Priors to Model Behavior: Structural Coverage in Tabular Foundation Models [15.974042867454365]
  We measure how broadly and repeatedly each synthetic prior reaches benchmark tasks using structural coverage and normalized density.<n>We find substantial differences across synthetic pretraining priors.<n>Stronger synthetic-to-benchmark support is generally associated with better relative model performance.
  arXiv  Detail & Related papers  (2026-09-07T01:27:45Z)
• A Controlled Counterexample to Strong Proxy-Based Explanations of OOD Performance: in a Fixed Pretraining-and-Probing Setup [0.9805949492148788]
  Task-agnostic structure proxies are often used to interpret why one pretraining corpus transfers better than another.<n>We test this requirement in a fixed pretraining-and-probing setup motivated by computationally bounded notions of learned structure.<n>A proxy for total learned structure can fail to track the task-relevant structure that drives OOD performance, even in a controlled setting.
  arXiv  Detail & Related papers  (2026-05-12T05:36:46Z)
• Decidable By Construction: Design-Time Verification for Trustworthy AI [0.0]
  A prevailing assumption in machine learning is that model correctness must be enforced after the fact.<n>We observe that the properties determining whether an AI model is numerically stable, computationally correct, or consistent with a physical domain do not necessarily demand post hoc enforcement.
  arXiv  Detail & Related papers  (2026-03-26T13:09:36Z)
• Tokenization, Fusion and Decoupling: Bridging the Granularity Mismatch Between Large Language Models and Knowledge Graphs [20.946228883628013]
  We propose KGT, a novel framework that uses dedicated entity tokens to enable efficient, full-space prediction.<n>We first introduce specialized tokenization to construct feature representations at the level of dedicated entity tokens.<n>We then fuse pre-trained structural and textual features into these unified embeddings via a relation-guided gating mechanism.
  arXiv  Detail & Related papers  (2026-02-26T07:20:40Z)
• Classifier-Augmented Generation for Structured Workflow Prediction [5.92079054629498]
  We propose a system that translates natural language descriptions into executables.<n>It automatically predicts both the structure and detailed configuration of the flow.<n>This is the first system with a detailed evaluation across stage prediction, edge layout, and property generation for natural-driven authoring.
  arXiv  Detail & Related papers  (2025-10-10T18:38:25Z)
• Synthetic bootstrapped pretraining [52.92577542049469]
  We introduce Synthetic Bootstrapped Pretraining (SBP), a language model (LM) pretraining procedure.<n>SBP first learns a model of relations between documents from the pretraining dataset and then leverages it to synthesize a vast new corpus for joint training.<n>We find SBP consistently improves upon a strong repetition baseline and delivers a significant fraction of performance improvement attainable by an oracle upper bound.
  arXiv  Detail & Related papers  (2025-09-17T22:28:27Z)
• AlphaFold Database Debiasing for Robust Inverse Folding [58.792020809180336]
  We introduce a Debiasing Structure AutoEncoder (DeSAE) that learns to reconstruct native-like conformations from intentionally corrupted backbone geometries.<n>At inference, applying DeSAE to AFDB structures produces debiased structures that significantly improve inverse folding performance.
  arXiv  Detail & Related papers  (2025-06-10T02:25:31Z)
• Supervised Pretraining for Material Property Prediction [0.36868085124383626]
  Self-supervised learning (SSL) offers a promising alternative by pretraining on large, unlabeled datasets to develop foundation models.<n>In this work, we propose supervised pretraining, where available class information serves as surrogate labels to guide learning.<n>To further enhance representation learning, we propose a graph-based augmentation technique that injects noise to improve robustness without structurally deforming material graphs.
  arXiv  Detail & Related papers  (2025-04-27T19:00:41Z)
• "Principal Components" Enable A New Language of Images [79.45806370905775]
  We introduce a novel visual tokenization framework that embeds a provable PCA-like structure into the latent token space.<n>Our approach achieves state-of-the-art reconstruction performance and enables better interpretability to align with the human vision system.
  arXiv  Detail & Related papers  (2025-03-11T17:59:41Z)
• Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality [55.88910947643436]
  Self-supervised pre-training is essential for handling vast quantities of unlabeled data in practice. HiDe-Prompt is an innovative approach that explicitly optimize the hierarchical components with an ensemble of task-specific prompts and statistics. Our experiments demonstrate the superior performance of HiDe-Prompt and its robustness to pre-training paradigms in continual learning.
  arXiv  Detail & Related papers  (2023-10-11T06:51:46Z)
• Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations [70.41385310930846]
  We present an end-to-end framework Structure-CLIP to enhance multi-modal structured representations. We use scene graphs to guide the construction of semantic negative examples, which results in an increased emphasis on learning structured representations. A Knowledge-Enhance (KEE) is proposed to leverage SGK as input to further enhance structured representations.
  arXiv  Detail & Related papers  (2023-05-06T03:57:05Z)
• CSNE: Conditional Signed Network Embedding [77.54225346953069]
  Signed networks encode positive and negative relations between entities such as friend/foe or trust/distrust. Existing embedding methods for sign prediction generally enforce different notions of status or balance theories in their optimization function. We introduce conditional signed network embedding (CSNE) Our probabilistic approach models structural information about the signs in the network separately from fine-grained detail.
  arXiv  Detail & Related papers  (2020-05-19T19:14:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.