TRON: Transformer Neural Network Acceleration with Non-Coherent Silicon
Photonics
- URL: http://arxiv.org/abs/2303.12914v1
- Date: Wed, 22 Mar 2023 21:09:49 GMT
- Title: TRON: Transformer Neural Network Acceleration with Non-Coherent Silicon
Photonics
- Authors: Salma Afifi, Febin Sunny, Mahdi Nikdast, Sudeep Pasricha
- Abstract summary: We propose the first silicon photonic hardware neural network accelerator called TRON for transformer-based models such as BERT, and Vision Transformers.
Our analysis demonstrates that TRON exhibits at least 14x better throughput and 8x better energy efficiency, in comparison to state-of-the-art transformer accelerators.
- Score: 4.616703548353372
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Transformer neural networks are rapidly being integrated into
state-of-the-art solutions for natural language processing (NLP) and computer
vision. However, the complex structure of these models creates challenges for
accelerating their execution on conventional electronic platforms. We propose
the first silicon photonic hardware neural network accelerator called TRON for
transformer-based models such as BERT, and Vision Transformers. Our analysis
demonstrates that TRON exhibits at least 14x better throughput and 8x better
energy efficiency, in comparison to state-of-the-art transformer accelerators.
Related papers
- ViT-LCA: A Neuromorphic Approach for Vision Transformers [0.0]
This paper introduces a novel model that combines vision transformers with the Locally Competitive Algorithm (LCA) to facilitate efficient neuromorphic deployment.
Our experiments show that ViT-LCA achieves higher accuracy on ImageNet-1K dataset while consuming significantly less energy than other spiking vision transformer counterparts.
arXiv Detail & Related papers (2024-10-31T18:41:30Z) - On the Design Space Between Transformers and Recursive Neural Nets [64.862738244735]
Continuous Recursive Neural Networks (CRvNN) and Neural Data Routers (NDR) are studied.
CRvNN pushes the boundaries of traditional RvNN, relaxing its discrete structure-wise composition and ending up with a Transformer-like structure.
NDR constrains the original Transformer to induce better structural inductive bias, ending up with a model that is close to CRvNN.
arXiv Detail & Related papers (2024-09-03T02:03:35Z) - ARTEMIS: A Mixed Analog-Stochastic In-DRAM Accelerator for Transformer Neural Networks [2.9699290794642366]
ARTEMIS is a mixed analog-stochastic in-DRAM accelerator for transformer models.
Our analysis indicates that ARTEMIS exhibits at least 3.0x speedup, 1.8x lower energy, and 1.9x better energy efficiency compared to GPU, TPU, CPU, and state-of-the-art PIM transformer hardware accelerators.
arXiv Detail & Related papers (2024-07-17T15:08:14Z) - EEGEncoder: Advancing BCI with Transformer-Based Motor Imagery Classification [11.687193535939798]
Brain-computer interfaces (BCIs) harness electroencephalographic signals for direct neural control of devices.
Traditional machine learning methods for EEG-based motor imagery (MI) classification encounter challenges such as manual feature extraction and susceptibility to noise.
This paper introduces EEGEncoder, a deep learning framework that employs modified transformers and TCNs to surmount these limitations.
arXiv Detail & Related papers (2024-04-23T09:51:24Z) - Accelerating Neural Networks for Large Language Models and Graph
Processing with Silicon Photonics [4.471962177124311]
Large language models (LLMs) and graph processing have emerged as transformative technologies for natural language processing (NLP), computer vision, and graph-structured data applications.
However, the complex structures of these models pose challenges for acceleration on conventional electronic platforms.
We describe novel hardware accelerators based on silicon photonics to accelerate transformer neural networks that are used in LLMs and graph neural networks for graph data processing.
arXiv Detail & Related papers (2024-01-12T20:32:38Z) - A Survey of Techniques for Optimizing Transformer Inference [3.6258657276072253]
Recent years have seen a phenomenal rise in performance and applications of transformer neural networks.
Transformer-based networks such as ChatGPT have impacted the lives of common men.
Researchers have proposed techniques to optimize transformer inference at all levels of abstraction.
arXiv Detail & Related papers (2023-07-16T08:50:50Z) - The Nuts and Bolts of Adopting Transformer in GANs [124.30856952272913]
We investigate the properties of Transformer in the generative adversarial network (GAN) framework for high-fidelity image synthesis.
Our study leads to a new alternative design of Transformers in GAN, a convolutional neural network (CNN)-free generator termed as STrans-G.
arXiv Detail & Related papers (2021-10-25T17:01:29Z) - Augmented Shortcuts for Vision Transformers [49.70151144700589]
We study the relationship between shortcuts and feature diversity in vision transformer models.
We present an augmented shortcut scheme, which inserts additional paths with learnable parameters in parallel on the original shortcuts.
Experiments conducted on benchmark datasets demonstrate the effectiveness of the proposed method.
arXiv Detail & Related papers (2021-06-30T09:48:30Z) - Transformers Solve the Limited Receptive Field for Monocular Depth
Prediction [82.90445525977904]
We propose TransDepth, an architecture which benefits from both convolutional neural networks and transformers.
This is the first paper which applies transformers into pixel-wise prediction problems involving continuous labels.
arXiv Detail & Related papers (2021-03-22T18:00:13Z) - Transformers in Vision: A Survey [101.07348618962111]
Transformers enable modeling long dependencies between input sequence elements and support parallel processing of sequence.
Transformers require minimal inductive biases for their design and are naturally suited as set-functions.
This survey aims to provide a comprehensive overview of the Transformer models in the computer vision discipline.
arXiv Detail & Related papers (2021-01-04T18:57:24Z) - A Survey on Visual Transformer [126.56860258176324]
Transformer is a type of deep neural network mainly based on the self-attention mechanism.
In this paper, we review these vision transformer models by categorizing them in different tasks and analyzing their advantages and disadvantages.
arXiv Detail & Related papers (2020-12-23T09:37:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.