Dimensionality Reduction in Sentence Transformer Vector Databases with Fast Fourier Transform

• URL: http://arxiv.org/abs/2404.06278v1
• Date: Tue, 9 Apr 2024 13:02:22 GMT
• Title: Dimensionality Reduction in Sentence Transformer Vector Databases with Fast Fourier Transform
• Authors: Vitaly Bulgakov, Alec Segal,
• Abstract summary: Dimensionality reduction in vector databases is pivotal for streamlining AI data management. This paper explores the benefits of reducing vector database dimensions, with a focus on computational efficiency and overcoming the curse of dimensionality. We introduce a novel application of Fast Fourier Transform (FFT) to dimensionality reduction, a method previously underexploited in this context.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dimensionality reduction in vector databases is pivotal for streamlining AI data management, enabling efficient storage, faster computation, and improved model performance. This paper explores the benefits of reducing vector database dimensions, with a focus on computational efficiency and overcoming the curse of dimensionality. We introduce a novel application of Fast Fourier Transform (FFT) to dimensionality reduction, a method previously underexploited in this context. By demonstrating its utility across various AI domains, including Retrieval-Augmented Generation (RAG) models and image processing, this FFT-based approach promises to improve data retrieval processes and enhance the efficiency and scalability of AI solutions. The incorporation of FFT may not only optimize operations in real-time processing and recommendation systems but also extend to advanced image processing techniques, where dimensionality reduction can significantly improve performance and analysis efficiency. This paper advocates for the broader adoption of FFT in vector database management, marking a significant stride towards addressing the challenges of data volume and complexity in AI research and applications. Unlike many existing approaches, we directly handle the embedding vectors produced by the model after processing a test input.

Related papers

• Multi-Dimensional Visual Data Recovery: Scale-Aware Tensor Modeling and Accelerated Randomized Computation [51.65236537605077]
  We propose a new type of network compression optimization technique, fully randomized tensor network compression (FCTN)<n>FCTN has significant advantages in correlation characterization and transpositional in algebra, and has notable achievements in multi-dimensional data processing and analysis.<n>We derive efficient algorithms with guarantees to solve the formulated models.
  arXiv  Detail & Related papers  (2026-02-13T14:56:37Z)
• GPT-FT: An Efficient Automated Feature Transformation Using GPT for Sequence Reconstruction and Performance Enhancement [18.888674282162032]
  Feature transformation plays a critical role in enhancing machine learning model performance by optimizing data representations.<n>Recent state-of-the-art approaches address this task as a continuous embedding optimization problem, converting discrete search into a learnable process.<n>We propose a novel framework that accomplishes automated feature transformation through four steps.
  arXiv  Detail & Related papers  (2025-08-28T14:21:08Z)
• An Efficient Data Reuse with Tile-Based Adaptive Stationary for Transformer Accelerators [0.0502254944841629]
  Transformer-based models have become the textitde facto backbone across many fields, such as computer vision and natural language processing. external memory access (EMA) for weight and activations becomes a critical bottleneck due to its significantly higher energy consumption compared to internal computations. We propose the Tile-based Adaptive Stationary scheme that selects the input or weight stationary in a tile, based on the input sequence length.
  arXiv  Detail & Related papers  (2025-03-25T13:29:58Z)
• PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection [68.8373788348678]
  Visual instruction tuning adapts pre-trained Multimodal Large Language Models to follow human instructions.<n>PRISM is the first training-free framework for efficient visual instruction selection.<n>It reduces the end-to-end time for data selection and model tuning to just 30% of conventional pipelines.
  arXiv  Detail & Related papers  (2025-02-17T18:43:41Z)
• TOAST: Transformer Optimization using Adaptive and Simple Transformations [40.311292704886235]
  We introduce TOAST, a framework that exploits redundancies to approximate entire transformer blocks with lightweight closed-form mappings.<n>Results show that large portions of transformer depth can be replaced by trivial functions, opening a new perspective on efficient foundation models.
  arXiv  Detail & Related papers  (2024-10-07T11:35:24Z)
• Transforming Image Super-Resolution: A ConvFormer-based Efficient Approach [58.57026686186709]
  We introduce the Convolutional Transformer layer (ConvFormer) and propose a ConvFormer-based Super-Resolution network (CFSR) CFSR inherits the advantages of both convolution-based and transformer-based approaches. Experiments demonstrate that CFSR strikes an optimal balance between computational cost and performance.
  arXiv  Detail & Related papers  (2024-01-11T03:08:00Z)
• Functional Graphical Models: Structure Enables Offline Data-Driven Optimization [111.28605744661638]
  We show how structure can enable sample-efficient data-driven optimization. We also present a data-driven optimization algorithm that infers the FGM structure itself.
  arXiv  Detail & Related papers  (2024-01-08T22:33:14Z)
• Variational Bayes image restoration with compressive autoencoders [4.879530644978008]
  Regularization of inverse problems is of paramount importance in computational imaging. In this work, we first propose to use compressive autoencoders instead of state-of-the-art generative models. As a second contribution, we introduce the Variational Bayes Latent Estimation (VBLE) algorithm.
  arXiv  Detail & Related papers  (2023-11-29T15:49:31Z)
• PREM: A Simple Yet Effective Approach for Node-Level Graph Anomaly Detection [65.24854366973794]
  Node-level graph anomaly detection (GAD) plays a critical role in identifying anomalous nodes from graph-structured data in domains such as medicine, social networks, and e-commerce. We introduce a simple method termed PREprocessing and Matching (PREM for short) to improve the efficiency of GAD. Our approach streamlines GAD, reducing time and memory consumption while maintaining powerful anomaly detection capabilities.
  arXiv  Detail & Related papers  (2023-10-18T02:59:57Z)
• Randomized Dimension Reduction with Statistical Guarantees [0.27195102129095]
  This thesis explores some of such algorithms for fast execution and efficient data utilization. We focus on learning algorithms with various incorporations of data augmentation that improve generalization and distributional provably. Specifically, Chapter 4 presents a sample complexity analysis for data augmentation consistency regularization.
  arXiv  Detail & Related papers  (2023-10-03T02:01:39Z)
• Analysis and Optimization of Wireless Federated Learning with Data Heterogeneity [72.85248553787538]
  This paper focuses on performance analysis and optimization for wireless FL, considering data heterogeneity, combined with wireless resource allocation. We formulate the loss function minimization problem, under constraints on long-term energy consumption and latency, and jointly optimize client scheduling, resource allocation, and the number of local training epochs (CRE) Experiments on real-world datasets demonstrate that the proposed algorithm outperforms other benchmarks in terms of the learning accuracy and energy consumption.
  arXiv  Detail & Related papers  (2023-08-04T04:18:01Z)
• Prescriptive PCA: Dimensionality Reduction for Two-stage Stochastic Optimization [1.1612308609123565]
  We develop a prescriptive dimensionality reduction framework that aims to minimize the degree of suboptimality in the optimization phase. For the case where the downstream optimization problem has an expected value objective, we show that prescriptive dimensionality reduction can be performed via solving a distributionally-robust optimization problem. Our approach significantly outperforms principal component analysis with real and synthetic data sets.
  arXiv  Detail & Related papers  (2023-06-04T00:50:35Z)
• Performance Embeddings: A Similarity-based Approach to Automatic Performance Optimization [71.69092462147292]
  Performance embeddings enable knowledge transfer of performance tuning between applications. We demonstrate this transfer tuning approach on case studies in deep neural networks, dense and sparse linear algebra compositions, and numerical weather prediction stencils.
  arXiv  Detail & Related papers  (2023-03-14T15:51:35Z)
• Transform Once: Efficient Operator Learning in Frequency Domain [69.74509540521397]
  We study deep neural networks designed to harness the structure in frequency domain for efficient learning of long-range correlations in space or time. This work introduces a blueprint for frequency domain learning through a single transform: transform once (T1)
  arXiv  Detail & Related papers  (2022-11-26T01:56:05Z)
• Resource-Efficient Invariant Networks: Exponential Gains by Unrolled Optimization [8.37077056358265]
  We propose a new computational primitive for building invariant networks based instead on optimization. We provide empirical and theoretical corroboration of the efficiency gains and soundness of our proposed method. We demonstrate its utility in constructing an efficient invariant network for a simple hierarchical object detection task.
  arXiv  Detail & Related papers  (2022-03-09T19:04:08Z)
• Recovery of Linear Components: Reduced Complexity Autoencoder Designs [0.951828574518325]
  We present an approach called Recovery of Linear Components (RLC), which serves as a middle ground between linear and non-linear dimensionality reduction techniques. With the aid of synthetic and real world case studies, we show that the RLC, when compared with an autoencoder of similar complexity, shows higher accuracy, similar to robustness to overfitting, and faster training times.
  arXiv  Detail & Related papers  (2020-12-14T14:08:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.