Towards Signal Processing In Large Language Models

• URL: http://arxiv.org/abs/2406.10254v1
• Date: Mon, 10 Jun 2024 13:51:52 GMT
• Title: Towards Signal Processing In Large Language Models
• Authors: Prateek Verma, Mert Pilanci,
• Abstract summary: This paper introduces the idea of applying signal processing inside a Large Language Model (LLM) We draw parallels between classical Fourier-Transforms and Fourier Transform-like learnable time-frequency representations. We show that for GPT-like architectures, our work achieves faster convergence and significantly increases performance.
• Score: 46.76681147411957
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces the idea of applying signal processing inside a Large Language Model (LLM). With the recent explosion of generative AI, our work can help bridge two fields together, namely the field of signal processing and large language models. We draw parallels between classical Fourier-Transforms and Fourier Transform-like learnable time-frequency representations for every intermediate activation signal of an LLM. Once we decompose every activation signal across tokens into a time-frequency representation, we learn how to filter and reconstruct them, with all components learned from scratch, to predict the next token given the previous context. We show that for GPT-like architectures, our work achieves faster convergence and significantly increases performance by adding a minuscule number of extra parameters when trained for the same epochs. We hope this work paves the way for algorithms exploring signal processing inside the signals found in neural architectures like LLMs and beyond.

Related papers

• WaveletGPT: Wavelets Meet Large Language Models [1.2328446298523066]
  Large Language Models (LLMs) have ushered in a new wave of artificial intelligence advancements. This paper infuses LLMs with traditional signal processing ideas, namely wavelets, during pre-training to take advantage of the structure. We achieve the same pre-training performance almost twice as fast in text, raw audio, and symbolic music.
  arXiv  Detail & Related papers  (2024-09-04T03:17:19Z)
• Content Adaptive Front End For Audio Signal Processing [2.8935588665357077]
  We propose a learnable content adaptive front end for audio signal processing. We pass each audio signal through a bank of convolutional filters, each giving a fixed-dimensional vector.
  arXiv  Detail & Related papers  (2023-03-18T16:09:10Z)
• Polynomial Neural Fields for Subband Decomposition and Manipulation [78.2401411189246]
  We propose a new class of neural fields called neural fields (PNFs) The key advantage of a PNF is that it can represent a signal as a composition of manipulable and interpretable components without losing the merits of neural fields. We empirically demonstrate that Fourier PNFs enable signal manipulation applications such as texture transfer and scale-space.
  arXiv  Detail & Related papers  (2023-02-09T18:59:04Z)
• Versatile Neural Processes for Learning Implicit Neural Representations [57.090658265140384]
  We propose Versatile Neural Processes (VNP), which largely increases the capability of approximating functions. Specifically, we introduce a bottleneck encoder that produces fewer and informative context tokens, relieving the high computational cost. We demonstrate the effectiveness of the proposed VNP on a variety of tasks involving 1D, 2D and 3D signals.
  arXiv  Detail & Related papers  (2023-01-21T04:08:46Z)
• Neural Fourier Filter Bank [18.52741992605852]
  We present a novel method to provide efficient and highly detailed reconstructions. Inspired by wavelets, we learn a neural field that decompose the signal both spatially and frequency-wise.
  arXiv  Detail & Related papers  (2022-12-04T03:45:08Z)
• Fourier Disentangled Space-Time Attention for Aerial Video Recognition [54.80846279175762]
  We present an algorithm, Fourier Activity Recognition (FAR), for UAV video activity recognition. Our formulation uses a novel Fourier object disentanglement method to innately separate out the human agent from the background. We have evaluated our approach on multiple UAV datasets including UAV Human RGB, UAV Human Night, Drone Action, and NEC Drone.
  arXiv  Detail & Related papers  (2022-03-21T01:24:53Z)
• Meta-Learning Sparse Implicit Neural Representations [69.15490627853629]
  Implicit neural representations are a promising new avenue of representing general signals. Current approach is difficult to scale for a large number of signals or a data set. We show that meta-learned sparse neural representations achieve a much smaller loss than dense meta-learned models.
  arXiv  Detail & Related papers  (2021-10-27T18:02:53Z)
• Large Scale Audio Understanding without Transformers/ Convolutions/ BERTs/ Mixers/ Attention/ RNNs or .... [4.594159253008448]
  This paper presents a way of doing large scale audio understanding without traditional state of the art neural architectures. Our approach does not have any convolutions, recurrence, attention, transformers or other approaches such as BERT. A classification head (a feed-forward layer), similar to the approach in SimCLR is trained on a learned representation.
  arXiv  Detail & Related papers  (2021-10-07T05:00:26Z)
• Modulated Periodic Activations for Generalizable Local Functional Representations [113.64179351957888]
  We present a new representation that generalizes to multiple instances and achieves state-of-the-art fidelity. Our approach produces general functional representations of images, videos and shapes, and achieves higher reconstruction quality than prior works that are optimized for a single signal.
  arXiv  Detail & Related papers  (2021-04-08T17:59:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.