Ares: Approximate Representations via Efficient Sparsification -- A Stateless Approach through Polynomial Homomorphism

• URL: http://arxiv.org/abs/2412.10623v1
• Date: Sat, 14 Dec 2024 00:05:43 GMT
• Title: Ares: Approximate Representations via Efficient Sparsification -- A Stateless Approach through Polynomial Homomorphism
• Authors: Dongfang Zhao,
• Abstract summary: We introduce a stateless compression framework that leverages limiting representations to achieve compact, interpretable and scalable data reduction. Our approach achieves high compression ratios without compromising reconstruction accuracy, all while maintaining simplicity and scalability.
• Score: 1.3824176915623292
• License:
• Abstract: The increasing prevalence of high-dimensional data demands efficient and scalable compression methods to support modern applications. However, existing techniques like PCA and Autoencoders often rely on auxiliary metadata or intricate architectures, limiting their practicality for streaming or infinite datasets. In this paper, we introduce a stateless compression framework that leverages polynomial representations to achieve compact, interpretable, and scalable data reduction. By eliminating the need for auxiliary data, our method supports direct algebraic operations in the compressed domain while minimizing error growth during computations. Through extensive experiments on synthetic and real-world datasets, we show that our approach achieves high compression ratios without compromising reconstruction accuracy, all while maintaining simplicity and scalability.

Related papers

• Theoretical Guarantees for Low-Rank Compression of Deep Neural Networks [5.582683296425384]
  Deep neural networks have achieved state-of-the-art performance across numerous applications. Low-rank approximation techniques offer a promising solution by reducing the size and complexity of these networks. We develop an analytical framework for data-driven post-training low-rank compression.
  arXiv  Detail & Related papers  (2025-02-04T23:10:13Z)
• A framework for compressing unstructured scientific data via serialization [2.5768995309704104]
  We present a general framework for compressing unstructured scientific data with known local connectivity. A common application is simulation data defined on arbitrary finite element meshes. The framework employs a greedy topology preserving reordering of original nodes which allows for seamless integration into existing data processing pipelines.
  arXiv  Detail & Related papers  (2024-10-10T15:53:35Z)
• Convex Distillation: Efficient Compression of Deep Networks via Convex Optimization [46.18363767705346]
  Deployment of large and complex convex networks on resource-constrained devices significant challenges due to their demands. We introduce novel distillation technique that efficiently compresses model via this model via this paper. Our approach enables performance comparable to original model without requiring any post-processing.
  arXiv  Detail & Related papers  (2024-10-09T06:04:52Z)
• Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models [56.00251589760559]
  Large language models (LLMs) can act as gradient priors in a zero-shot setting. We introduce LM-GC, a novel method that integrates LLMs with arithmetic coding. Experiments indicate that LM-GC surpasses existing state-of-the-art lossless compression methods.
  arXiv  Detail & Related papers  (2024-09-26T13:38:33Z)
• Hierarchical Features Matter: A Deep Exploration of GAN Priors for Improved Dataset Distillation [51.44054828384487]
  We propose a novel parameterization method dubbed Hierarchical Generative Latent Distillation (H-GLaD) This method systematically explores hierarchical layers within the generative adversarial networks (GANs) In addition, we introduce a novel class-relevant feature distance metric to alleviate the computational burden associated with synthetic dataset evaluation.
  arXiv  Detail & Related papers  (2024-06-09T09:15:54Z)
• Sparse $L^1$-Autoencoders for Scientific Data Compression [0.0]
  We introduce effective data compression methods by developing autoencoders using high dimensional latent spaces that are $L1$-regularized. We show how these information-rich latent spaces can be used to mitigate blurring and other artifacts to obtain highly effective data compression methods for scientific data.
  arXiv  Detail & Related papers  (2024-05-23T07:48:00Z)
• Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth [83.15263499262824]
  We prove that gradient descent converges to a solution that completely disregards the sparse structure of the input. We show how to improve upon Gaussian performance for the compression of sparse data by adding a denoising function to a shallow architecture. We validate our findings on image datasets, such as CIFAR-10 and MNIST.
  arXiv  Detail & Related papers  (2024-02-07T16:32:29Z)
• Rethinking Compression: Reduced Order Modelling of Latent Features in Large Language Models [9.91972450276408]
  This paper introduces an innovative approach for the parametric and practical compression of Large Language Models (LLMs) based on reduced order modelling. Our method represents a significant advancement in model compression by leveraging matrix decomposition, demonstrating superior efficacy compared to the prevailing state-of-the-art structured pruning method.
  arXiv  Detail & Related papers  (2023-12-12T07:56:57Z)
• Linear Self-Attention Approximation via Trainable Feedforward Kernel [77.34726150561087]
  In pursuit of faster computation, Efficient Transformers demonstrate an impressive variety of approaches. We aim to expand the idea of trainable kernel methods to approximate the self-attention mechanism of the Transformer architecture.
  arXiv  Detail & Related papers  (2022-11-08T08:14:11Z)
• Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression [56.83861738731913]
  Deep Neural Network (DNN) models are essential for practical applications, especially for resource limited devices. Previous unstructured or structured weight pruning methods can hardly truly accelerate inference. We propose a generalized weight unification framework at a hardware compatible micro-structured level to achieve high amount of compression and acceleration.
  arXiv  Detail & Related papers  (2021-06-15T17:22:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.