VN-Transformer: Rotation-Equivariant Attention for Vector Neurons
- URL: http://arxiv.org/abs/2206.04176v1
- Date: Wed, 8 Jun 2022 21:48:47 GMT
- Title: VN-Transformer: Rotation-Equivariant Attention for Vector Neurons
- Authors: Serge Assaad, Carlton Downey, Rami Al-Rfou, Nigamaa Nayakanti, Ben
Sapp
- Abstract summary: We introduce a novel "VN-Transformer" architecture to address several shortcomings of the current VN models.
We apply our VN-Transformer to 3D shape classification and motion forecasting with compelling results.
- Score: 13.027392571994204
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Rotation equivariance is a desirable property in many practical applications
such as motion forecasting and 3D perception, where it can offer benefits like
sample efficiency, better generalization, and robustness to input
perturbations. Vector Neurons (VN) is a recently developed framework offering a
simple yet effective approach for deriving rotation-equivariant analogs of
standard machine learning operations by extending one-dimensional scalar
neurons to three-dimensional "vector neurons." We introduce a novel
"VN-Transformer" architecture to address several shortcomings of the current VN
models. Our contributions are: $(i)$ we derive a rotation-equivariant attention
mechanism which eliminates the need for the heavy feature preprocessing
required by the original Vector Neurons models; $(ii)$ we extend the VN
framework to support non-spatial attributes, expanding the applicability of
these models to real-world datasets; $(iii)$ we derive a rotation-equivariant
mechanism for multi-scale reduction of point-cloud resolution, greatly speeding
up inference and training; $(iv)$ we show that small tradeoffs in equivariance
($\epsilon$-approximate equivariance) can be used to obtain large improvements
in numerical stability and training robustness on accelerated hardware, and we
bound the propagation of equivariance violations in our models. Finally, we
apply our VN-Transformer to 3D shape classification and motion forecasting with
compelling results.
Related papers
- Equivariant Graph Neural Operator for Modeling 3D Dynamics [148.98826858078556]
We propose Equivariant Graph Neural Operator (EGNO) to directly models dynamics as trajectories instead of just next-step prediction.
EGNO explicitly learns the temporal evolution of 3D dynamics where we formulate the dynamics as a function over time and learn neural operators to approximate it.
Comprehensive experiments in multiple domains, including particle simulations, human motion capture, and molecular dynamics, demonstrate the significantly superior performance of EGNO against existing methods.
arXiv Detail & Related papers (2024-01-19T21:50:32Z) - Negative Feedback Training: A Novel Concept to Improve Robustness of NVCIM DNN Accelerators [11.832487701641723]
Non-volatile memory (NVM) devices excel in energy efficiency and latency when performing Deep Neural Network (DNN) inference.
We propose a novel training concept: Negative Feedback Training (NFT) leveraging the multi-scale noisy information captured from network.
Our methods outperform existing state-of-the-art methods with up to a 46.71% improvement in inference accuracy.
arXiv Detail & Related papers (2023-05-23T22:56:26Z) - E($3$) Equivariant Graph Neural Networks for Particle-Based Fluid
Mechanics [2.1401663582288144]
We demonstrate that equivariant graph neural networks have the potential to learn more accurate dynamic-interaction models.
We benchmark two well-studied fluid flow systems, namely the 3D decaying Taylor-Green vortex and the 3D reverse Poiseuille flow.
arXiv Detail & Related papers (2023-03-31T21:56:35Z) - Category-Level 6D Object Pose Estimation with Flexible Vector-Based
Rotation Representation [51.67545893892129]
We propose a novel 3D graph convolution based pipeline for category-level 6D pose and size estimation from monocular RGB-D images.
We first design an orientation-aware autoencoder with 3D graph convolution for latent feature learning.
Then, to efficiently decode the rotation information from the latent feature, we design a novel flexible vector-based decomposable rotation representation.
arXiv Detail & Related papers (2022-12-09T02:13:43Z) - REMuS-GNN: A Rotation-Equivariant Model for Simulating Continuum
Dynamics [0.0]
We introduce REMuS-GNN, a rotation-equivariant multi-scale model for simulating continuum dynamical systems.
We demonstrate and evaluate this method on the incompressible flow around elliptical cylinders.
arXiv Detail & Related papers (2022-05-05T16:20:37Z) - Equivariant vector field network for many-body system modeling [65.22203086172019]
Equivariant Vector Field Network (EVFN) is built on a novel equivariant basis and the associated scalarization and vectorization layers.
We evaluate our method on predicting trajectories of simulated Newton mechanics systems with both full and partially observed data.
arXiv Detail & Related papers (2021-10-26T14:26:25Z) - Frame Averaging for Invariant and Equivariant Network Design [50.87023773850824]
We introduce Frame Averaging (FA), a framework for adapting known (backbone) architectures to become invariant or equivariant to new symmetry types.
We show that FA-based models have maximal expressive power in a broad setting.
We propose a new class of universal Graph Neural Networks (GNNs), universal Euclidean motion invariant point cloud networks, and Euclidean motion invariant Message Passing (MP) GNNs.
arXiv Detail & Related papers (2021-10-07T11:05:23Z) - Orthogonal Graph Neural Networks [53.466187667936026]
Graph neural networks (GNNs) have received tremendous attention due to their superiority in learning node representations.
stacking more convolutional layers significantly decreases the performance of GNNs.
We propose a novel Ortho-GConv, which could generally augment the existing GNN backbones to stabilize the model training and improve the model's generalization performance.
arXiv Detail & Related papers (2021-09-23T12:39:01Z) - SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks [71.55002934935473]
We introduce the SE(3)-Transformer, a variant of the self-attention module for 3D point clouds and graphs, which is equivariant under continuous 3D roto-translations.
We evaluate our model on a toy N-body particle simulation dataset, showcasing the robustness of the predictions under rotations of the input.
arXiv Detail & Related papers (2020-06-18T13:23:01Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.