Locality-Sensitive Hashing-Based Efficient Point Transformer with Applications in High-Energy Physics

• URL: http://arxiv.org/abs/2402.12535v2
• Date: Wed, 5 Jun 2024 16:57:00 GMT
• Title: Locality-Sensitive Hashing-Based Efficient Point Transformer with Applications in High-Energy Physics
• Authors: Siqi Miao, Zhiyuan Lu, Mia Liu, Javier Duarte, Pan Li,
• Abstract summary: This study introduces a novel transformer model optimized for large-scale point cloud processing. Our model integrates local inductive bias and achieves near-linear complexity with hardware-friendly regular operations. Our findings highlight the superiority of using locality-sensitive hashing (LSH), especially OR & AND-construction LSH, in kernel approximation for large-scale point cloud data.
• Score: 11.182510067821745
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study introduces a novel transformer model optimized for large-scale point cloud processing in scientific domains such as high-energy physics (HEP) and astrophysics. Addressing the limitations of graph neural networks and standard transformers, our model integrates local inductive bias and achieves near-linear complexity with hardware-friendly regular operations. One contribution of this work is the quantitative analysis of the error-complexity tradeoff of various sparsification techniques for building efficient transformers. Our findings highlight the superiority of using locality-sensitive hashing (LSH), especially OR & AND-construction LSH, in kernel approximation for large-scale point cloud data with local inductive bias. Based on this finding, we propose LSH-based Efficient Point Transformer (HEPT), which combines E$^2$LSH with OR & AND constructions and is built upon regular computations. HEPT demonstrates remarkable performance on two critical yet time-consuming HEP tasks, significantly outperforming existing GNNs and transformers in accuracy and computational speed, marking a significant advancement in geometric deep learning and large-scale scientific data processing. Our code is available at https://github.com/Graph-COM/HEPT.

Related papers

• Liquid Fourier Latent Dynamics Networks for fast GPU-based numerical simulations in computational cardiology [0.0]
  We propose an extension of Latent Dynamics Networks (LDNets) to create parameterized space-time surrogate models for multiscale and multiphysics sets of highly nonlinear differential equations on complex geometries. LFLDNets employ a neurologically-inspired, sparse liquid neural network for temporal dynamics, relaxing the requirement of a numerical solver for time advancement and leading to superior performance in terms of parameters, accuracy, efficiency and learned trajectories.
  arXiv  Detail & Related papers  (2024-08-19T09:14:25Z)
• AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer [54.713778961605115]
  Vision Transformer (ViT) has become one of the most prevailing fundamental backbone networks in the computer vision community. We propose a novel non-uniform quantizer, dubbed the Adaptive Logarithm AdaLog (AdaLog) quantizer.
  arXiv  Detail & Related papers  (2024-07-17T18:38:48Z)
• Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysis [63.66763657191476]
  We show that efficient numerical training and inference algorithms as low-rank computation have impressive performance for learning Transformer-based adaption. We analyze how magnitude-based models affect generalization while improving adaption. We conclude that proper magnitude-based has a slight on the testing performance.
  arXiv  Detail & Related papers  (2024-06-24T23:00:58Z)
• Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators [12.165876595927452]
  Universal Physics Transformers (UPTs) are efficient and unified learning paradigm for a wide range of problems. UPTs operate without grid- or particle-based latent meshes, enabling flexibility across structures and particles. We demonstrate diverse applicability and efficacy of UPTs in mesh-based fluid simulations, and steady-state Reynolds averaged Navier-Stokes simulations.
  arXiv  Detail & Related papers  (2024-02-19T18:52:13Z)
• TransAxx: Efficient Transformers with Approximate Computing [4.347898144642257]
  Vision Transformer (ViT) models have shown to be very competitive and often become a popular alternative to Convolutional Neural Networks (CNNs) We propose TransAxx, a framework based on the popular PyTorch library that enables fast inherent support for approximate arithmetic. Our approach uses a Monte Carlo Tree Search (MCTS) algorithm to efficiently search the space of possible configurations.
  arXiv  Detail & Related papers  (2024-02-12T10:16:05Z)
• A Hierarchical Spatial Transformer for Massive Point Samples in Continuous Space [11.074768589778934]
  Existing transformers are mostly designed for sequences (texts or time series), images or videos, and graphs. This paper proposes a novel transformer model for massive (up to a million) point samples in continuous space.
  arXiv  Detail & Related papers  (2023-11-08T02:54:19Z)
• Hyperparameter Optimization of Generative Adversarial Network Models for High-Energy Physics Simulations [0.0]
  The Generative Adversarial Network (GAN) is a powerful tool that can generate high-fidelity synthesized data by learning. This work provides the first insights into efficiently tuning GANs for Large Hadron Collider data.
  arXiv  Detail & Related papers  (2022-08-12T20:39:57Z)
• Your Transformer May Not be as Powerful as You Expect [88.11364619182773]
  We mathematically analyze the power of RPE-based Transformers regarding whether the model is capable of approximating any continuous sequence-to-sequence functions. We present a negative result by showing there exist continuous sequence-to-sequence functions that RPE-based Transformers cannot approximate no matter how deep and wide the neural network is. We develop a novel attention module, called Universal RPE-based (URPE) Attention, which satisfies the conditions.
  arXiv  Detail & Related papers  (2022-05-26T14:51:30Z)
• Inducing Gaussian Process Networks [80.40892394020797]
  We propose inducing Gaussian process networks (IGN), a simple framework for simultaneously learning the feature space as well as the inducing points. The inducing points, in particular, are learned directly in the feature space, enabling a seamless representation of complex structured domains. We report on experimental results for real-world data sets showing that IGNs provide significant advances over state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-04-21T05:27:09Z)
• Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding [63.539333383965726]
  We propose a novel way to accelerate attention calculation for Transformers with relative positional encoding (RPE) Based upon the observation that relative positional encoding forms a Toeplitz matrix, we mathematically show that kernelized attention with RPE can be calculated efficiently using Fast Fourier Transform (FFT)
  arXiv  Detail & Related papers  (2021-06-23T17:51:26Z)
• Efficient pre-training objectives for Transformers [84.64393460397471]
  We study several efficient pre-training objectives for Transformers-based models. We prove that eliminating the MASK token and considering the whole output during the loss are essential choices to improve performance.
  arXiv  Detail & Related papers  (2021-04-20T00:09:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.