Crystalformer: Infinitely Connected Attention for Periodic Structure Encoding
- URL: http://arxiv.org/abs/2403.11686v1
- Date: Mon, 18 Mar 2024 11:37:42 GMT
- Title: Crystalformer: Infinitely Connected Attention for Periodic Structure Encoding
- Authors: Tatsunori Taniai, Ryo Igarashi, Yuta Suzuki, Naoya Chiba, Kotaro Saito, Yoshitaka Ushiku, Kanta Ono,
- Abstract summary: Predicting physical properties of materials from their crystal structures is a fundamental problem in materials science.
We show that crystal structures are infinitely repeating, periodic arrangements of atoms, whose fully connected attention results in infinitely connected attention.
We propose a simple yet effective Transformer-based encoder architecture for crystal structures called Crystalformer.
- Score: 10.170537065646323
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Predicting physical properties of materials from their crystal structures is a fundamental problem in materials science. In peripheral areas such as the prediction of molecular properties, fully connected attention networks have been shown to be successful. However, unlike these finite atom arrangements, crystal structures are infinitely repeating, periodic arrangements of atoms, whose fully connected attention results in infinitely connected attention. In this work, we show that this infinitely connected attention can lead to a computationally tractable formulation, interpreted as neural potential summation, that performs infinite interatomic potential summations in a deeply learned feature space. We then propose a simple yet effective Transformer-based encoder architecture for crystal structures called Crystalformer. Compared to an existing Transformer-based model, the proposed model requires only 29.4% of the number of parameters, with minimal modifications to the original Transformer architecture. Despite the architectural simplicity, the proposed method outperforms state-of-the-art methods for various property regression tasks on the Materials Project and JARVIS-DFT datasets.
Related papers
- Generative Inverse Design of Crystal Structures via Diffusion Models with Transformers [1.2289361708127877]
New inorganic materials with promising properties pose a critical challenge, both scientifically and for industrial applications.
Discovery of new inorganic materials with promising properties poses a critical challenge, both scientifically and for industrial applications.
In this study, we explore a new type of diffusion model for the generative inverse design of crystal structures, with a backbone based on a Transformer architecture.
arXiv Detail & Related papers (2024-06-13T16:03:15Z) - Complete and Efficient Graph Transformers for Crystal Material Property Prediction [53.32754046881189]
Crystal structures are characterized by atomic bases within a primitive unit cell that repeats along a regular lattice throughout 3D space.
We introduce a novel approach that utilizes the periodic patterns of unit cells to establish the lattice-based representation for each atom.
We propose ComFormer, a SE(3) transformer designed specifically for crystalline materials.
arXiv Detail & Related papers (2024-03-18T15:06:37Z) - Universal Machine Learning Kohn-Sham Hamiltonian for Materials [5.189794091596078]
This study introduces a universal electronic Hamiltonian model trained on Hamiltonian matrices.
We demonstrate its generality in predicting electronic structures across the whole periodic table.
By offering a reliable framework for computing electronic properties, this universal Hamiltonian model lays the groundwork for advancements in diverse fields.
arXiv Detail & Related papers (2024-02-14T15:38:56Z) - Scalable Diffusion for Materials Generation [99.71001883652211]
We develop a unified crystal representation that can represent any crystal structure (UniMat)
UniMat can generate high fidelity crystal structures from larger and more complex chemical systems.
We propose additional metrics for evaluating generative models of materials.
arXiv Detail & Related papers (2023-10-18T15:49:39Z) - Latent Conservative Objective Models for Data-Driven Crystal Structure
Prediction [62.36797874900395]
In computational chemistry, crystal structure prediction is an optimization problem.
One approach to tackle this problem involves building simulators based on density functional theory (DFT) followed by running search in simulation.
We show that our approach, dubbed LCOMs (latent conservative objective models), performs comparably to the best current approaches in terms of success rate of structure prediction.
arXiv Detail & Related papers (2023-10-16T04:35:44Z) - Crystal-GFN: sampling crystals with desirable properties and constraints [103.79058968784163]
We introduce Crystal-GFN, a generative model of crystal structures that sequentially samples structural properties of crystalline materials.
In this paper, we use as objective the formation energy per atom of a crystal structure predicted by a new proxy machine learning model trained on MatBench.
The results demonstrate that Crystal-GFN is able to sample highly diverse crystals with low (median -3.1 eV/atom) predicted formation energy.
arXiv Detail & Related papers (2023-10-07T21:36:55Z) - Molecular Geometry-aware Transformer for accurate 3D Atomic System
modeling [51.83761266429285]
We propose a novel Transformer architecture that takes nodes (atoms) and edges (bonds and nonbonding atom pairs) as inputs and models the interactions among them.
Moleformer achieves state-of-the-art on the initial state to relaxed energy prediction of OC20 and is very competitive in QM9 on predicting quantum chemical properties.
arXiv Detail & Related papers (2023-02-02T03:49:57Z) - Neural Structure Fields with Application to Crystal Structure
Autoencoders [10.680545976155173]
We propose neural structure fields (NeSF) as an accurate and practical approach for representing crystal structures using neural networks.
NeSF overcomes the tradeoff between spatial resolution and computational complexity and can represent any crystal structure.
We propose an autoencoder of crystal structures that can recover various crystal structures, such as those of perovskite structure materials and cuprate superconductors.
arXiv Detail & Related papers (2022-12-08T16:41:41Z) - Atomic structure generation from reconstructing structural fingerprints [1.2128971613239876]
We present an end-to-end structure generation approach using atom-centered symmetry functions as the representation and conditional variational autoencoders as the generative model.
We are able to successfully generate novel and valid atomic structures of sub-nanometer Pt nanoparticles as a proof of concept.
arXiv Detail & Related papers (2022-07-27T00:42:59Z) - Geometric Transformer for End-to-End Molecule Properties Prediction [92.28929858529679]
We introduce a Transformer-based architecture for molecule property prediction, which is able to capture the geometry of the molecule.
We modify the classical positional encoder by an initial encoding of the molecule geometry, as well as a learned gated self-attention mechanism.
arXiv Detail & Related papers (2021-10-26T14:14:40Z) - An invertible crystallographic representation for general inverse design
of inorganic crystals with targeted properties [10.853822721106205]
We present a framework capable of general inverse design (not limited to a given set of elements or crystal structures)
The framework generates new crystals with user-defined formation energies, bandgap, thermoelectric (TE) power factor, and combinations thereof.
Results represent a significant step toward property-driven general inverse design using generative models.
arXiv Detail & Related papers (2020-05-15T15:58:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.