A Framework of SO(3)-equivariant Non-linear Representation Learning and its Application to Electronic-Structure Hamiltonian Prediction

• URL: http://arxiv.org/abs/2405.05722v4
• Date: Tue, 15 Oct 2024 03:08:29 GMT
• Title: A Framework of SO(3)-equivariant Non-linear Representation Learning and its Application to Electronic-Structure Hamiltonian Prediction
• Authors: Shi Yin, Xinyang Pan, Fengyan Wang, Lixin He,
• Abstract summary: We propose a theoretical and a methodological framework to address a critical challenge in applying deep learning to physical systems. Inspired by covariant theory in physics, we present a solution by exploring the mathematical relationships between SO(3)-invariant and SO(3)-equivariant quantities. We show that our method boosts Hamiltonian prediction accuracy by up to 40% and enhances downstream physical quantities, such as occupied orbital energy, by a maximum of 76%.
• Score: 1.8982950873008362
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose both a theoretical and a methodological framework to address a critical challenge in applying deep learning to physical systems: the reconciliation of non-linear expressiveness with SO(3)-equivariance in predictions of SO(3)-equivariant quantities. Inspired by covariant theory in physics, we present a solution by exploring the mathematical relationships between SO(3)-invariant and SO(3)-equivariant quantities and their representations. We first construct theoretical SO(3)-invariant quantities derived from the SO(3)-equivariant regression targets, and use these invariant quantities as supervisory labels to guide the learning of high-quality SO(3)-invariant features. Given that SO(3)-invariance is preserved under non-linear operations, the encoding process for invariant features can extensively utilize non-linear mappings, thereby fully capturing the non-linear patterns inherent in physical systems. Building on this, we propose a gradient-based mechanism to induce SO(3)-equivariant encodings of various degrees from the learned SO(3)-invariant features. This mechanism can incorporate non-linear expressive capabilities into SO(3)-equivariant representations, while theoretically preserving their equivariant properties as we prove, establishing a strong foundation for regressing complex SO(3)-equivariant targets. We apply our theory and method to the electronic-structure Hamiltonian prediction tasks, experimental results on eight benchmark databases covering multiple types of systems and challenging scenarios show substantial improvements on the state-of-the-art prediction accuracy of deep learning paradigm. Our method boosts Hamiltonian prediction accuracy by up to 40% and enhances downstream physical quantities, such as occupied orbital energy, by a maximum of 76%.

Related papers

• Efficient and Scalable Density Functional Theory Hamiltonian Prediction through Adaptive Sparsity [11.415146682472127]
  We introduce an efficient and scalable equivariant network that incorporates adaptive sparsity into Hamiltonian prediction. We develop a Three-phase Sparsity Scheduler, ensuring stable convergence and achieving high performance at sparsity rates of up to 70 percent. Beyond Hamiltonian prediction, the proposed sparsification techniques also hold significant potential for improving the efficiency and scalability of other SE(3) equivariant networks.
  arXiv  Detail & Related papers  (2025-02-03T09:04:47Z)
• Harmonizing SO(3)-Equivariance with Neural Expressiveness: a Hybrid Deep Learning Framework Oriented to the Prediction of Electronic Structure Hamiltonian [36.13416266854978]
  HarmoSE is a two-stage cascaded regression framework for deep learning. First stage predicts Hamiltonians with abundant SO(3)-equivariant features extracted. Second stage refines the first stage's output as a fine-grained prediction of Hamiltonians.
  arXiv  Detail & Related papers  (2024-01-01T12:57:15Z)
• Stress representations for tensor basis neural networks: alternative formulations to Finger-Rivlin-Ericksen [0.0]
  We survey a variety of tensor neural network models for modeling hyperelastic deformation materials in a finite context. We compare potential-based and coefficient-based approaches, as well as different calibration techniques. Nine variants are tested against both noisy and noiseless datasets for three different materials.
  arXiv  Detail & Related papers  (2023-08-21T23:28:26Z)
• Data-driven Nonlinear Parametric Model Order Reduction Framework using Deep Hierarchical Variational Autoencoder [5.521324490427243]
  Data-driven parametric model order reduction (MOR) method using a deep artificial neural network is proposed. LSH-VAE is capable of performing nonlinear MOR for the parametric of a nonlinear dynamic system with a significant number of degrees of freedom.
  arXiv  Detail & Related papers  (2023-07-10T02:44:53Z)
• Stable and Consistent Prediction of 3D Characteristic Orientation via Invariant Residual Learning [42.44798841872727]
  We introduce a novel method to decouple the shape geometry and semantics of the input point cloud to achieve both stability and consistency. In experiments, the proposed method not only demonstrates superior stability and consistency but also exhibits state-of-the-art performances.
  arXiv  Detail & Related papers  (2023-06-20T09:29:03Z)
• Structure-Preserving Learning Using Gaussian Processes and Variational Integrators [62.31425348954686]
  We propose the combination of a variational integrator for the nominal dynamics of a mechanical system and learning residual dynamics with Gaussian process regression. We extend our approach to systems with known kinematic constraints and provide formal bounds on the prediction uncertainty.
  arXiv  Detail & Related papers  (2021-12-10T11:09:29Z)
• A Variational Inference Approach to Inverse Problems with Gamma Hyperpriors [60.489902135153415]
  This paper introduces a variational iterative alternating scheme for hierarchical inverse problems with gamma hyperpriors. The proposed variational inference approach yields accurate reconstruction, provides meaningful uncertainty quantification, and is easy to implement.
  arXiv  Detail & Related papers  (2021-11-26T06:33:29Z)
• Equivariant vector field network for many-body system modeling [65.22203086172019]
  Equivariant Vector Field Network (EVFN) is built on a novel equivariant basis and the associated scalarization and vectorization layers. We evaluate our method on predicting trajectories of simulated Newton mechanics systems with both full and partially observed data.
  arXiv  Detail & Related papers  (2021-10-26T14:26:25Z)
• A deep learning driven pseudospectral PCE based FFT homogenization algorithm for complex microstructures [68.8204255655161]
  It is shown that the proposed method is able to predict central moments of interest while being magnitudes faster to evaluate than traditional approaches. It is shown, that the proposed method is able to predict central moments of interest while being magnitudes faster to evaluate than traditional approaches.
  arXiv  Detail & Related papers  (2021-10-26T07:02:14Z)
• Nonlinearities in Steerable SO(2)-Equivariant CNNs [7.552100672006172]
  We apply harmonic distortion analysis to illuminate the effect of nonlinearities on representations of SO(2). We develop a novel FFT-based algorithm for computing representations of non-linearly transformed activations. In experiments with 2D and 3D data, we obtain results that compare favorably to the state-of-the-art in terms of accuracy while continuous symmetry and exact equivariance.
  arXiv  Detail & Related papers  (2021-09-14T17:53:45Z)
• Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms [71.62575565990502]
  We prove that the generalization error of an optimization algorithm can be bounded on the complexity' of the fractal structure that underlies its generalization measure. We further specialize our results to specific problems (e.g., linear/logistic regression, one hidden/layered neural networks) and algorithms.
  arXiv  Detail & Related papers  (2021-06-09T08:05:36Z)
• Neural Dynamic Mode Decomposition for End-to-End Modeling of Nonlinear Dynamics [49.41640137945938]
  We propose a neural dynamic mode decomposition for estimating a lift function based on neural networks. With our proposed method, the forecast error is backpropagated through the neural networks and the spectral decomposition. Our experiments demonstrate the effectiveness of our proposed method in terms of eigenvalue estimation and forecast performance.
  arXiv  Detail & Related papers  (2020-12-11T08:34:26Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)
• Learning Partially Known Stochastic Dynamics with Empirical PAC Bayes [12.44342023476206]
  This paper presents a recipe to improve the prediction accuracy of such models in three steps. We observe in our experiments that this recipe effectively translates partial and noisy prior knowledge into an improved model fit.
  arXiv  Detail & Related papers  (2020-06-17T14:47:06Z)
• Multiplicative noise and heavy tails in stochastic optimization [62.993432503309485]
  empirical optimization is central to modern machine learning, but its role in its success is still unclear. We show that it commonly arises in parameters of discrete multiplicative noise due to variance. A detailed analysis is conducted in which we describe on key factors, including recent step size, and data, all exhibit similar results on state-of-the-art neural network models.
  arXiv  Detail & Related papers  (2020-06-11T09:58:01Z)
• On dissipative symplectic integration with applications to gradient-based optimization [77.34726150561087]
  We propose a geometric framework in which discretizations can be realized systematically. We show that a generalization of symplectic to nonconservative and in particular dissipative Hamiltonian systems is able to preserve rates of convergence up to a controlled error.
  arXiv  Detail & Related papers  (2020-04-15T00:36:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.