Frequency Domain-based Dataset Distillation
- URL: http://arxiv.org/abs/2311.08819v1
- Date: Wed, 15 Nov 2023 09:46:30 GMT
- Title: Frequency Domain-based Dataset Distillation
- Authors: Donghyeok Shin, Seungjae Shin, Il-Chul Moon
- Abstract summary: We present FreD, a novel parameterization method for dataset distillation.
FreD employs frequency-based transforms to optimize the frequency representations of each data instance.
- Score: 17.02955182740882
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: This paper presents FreD, a novel parameterization method for dataset
distillation, which utilizes the frequency domain to distill a small-sized
synthetic dataset from a large-sized original dataset. Unlike conventional
approaches that focus on the spatial domain, FreD employs frequency-based
transforms to optimize the frequency representations of each data instance. By
leveraging the concentration of spatial domain information on specific
frequency components, FreD intelligently selects a subset of frequency
dimensions for optimization, leading to a significant reduction in the required
budget for synthesizing an instance. Through the selection of frequency
dimensions based on the explained variance, FreD demonstrates both theoretical
and empirical evidence of its ability to operate efficiently within a limited
budget, while better preserving the information of the original dataset
compared to conventional parameterization methods. Furthermore, based on the
orthogonal compatibility of FreD with existing methods, we confirm that FreD
consistently improves the performances of existing distillation methods over
the evaluation scenarios with different benchmark datasets. We release the code
at https://github.com/sdh0818/FreD.
Related papers
- LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning [47.77830360814755]
Location-aware Cosine Adaptation (LoCA) is a novel frequency-domain parameter-efficient fine-tuning method based on Discrete inverse Cosine Transform (iDCT)
Our analysis reveals that frequency-domain approximation with carefully selected frequency components can surpass the expressivity of traditional low-rank-based methods.
Experiments on diverse language and vision fine-tuning tasks demonstrate that LoCA offers enhanced parameter efficiency while maintains computational feasibility comparable to low-rank-based methods.
arXiv Detail & Related papers (2025-02-05T04:14:34Z) - Breaking Determinism: Fuzzy Modeling of Sequential Recommendation Using Discrete State Space Diffusion Model [66.91323540178739]
Sequential recommendation (SR) aims to predict items that users may be interested in based on their historical behavior.
We revisit SR from a novel information-theoretic perspective and find that sequential modeling methods fail to adequately capture randomness and unpredictability of user behavior.
Inspired by fuzzy information processing theory, this paper introduces the fuzzy sets of interaction sequences to overcome the limitations and better capture the evolution of users' real interests.
arXiv Detail & Related papers (2024-10-31T14:52:01Z) - Hierarchical Features Matter: A Deep Exploration of GAN Priors for Improved Dataset Distillation [51.44054828384487]
We propose a novel parameterization method dubbed Hierarchical Generative Latent Distillation (H-GLaD)
This method systematically explores hierarchical layers within the generative adversarial networks (GANs)
In addition, we introduce a novel class-relevant feature distance metric to alleviate the computational burden associated with synthetic dataset evaluation.
arXiv Detail & Related papers (2024-06-09T09:15:54Z) - State-Free Inference of State-Space Models: The Transfer Function Approach [132.83348321603205]
State-free inference does not incur any significant memory or computational cost with an increase in state size.
We achieve this using properties of the proposed frequency domain transfer function parametrization.
We report improved perplexity in language modeling over a long convolutional Hyena baseline.
arXiv Detail & Related papers (2024-05-10T00:06:02Z) - Supervised low-rank semi-nonnegative matrix factorization with frequency regularization for forecasting spatio-temporal data [6.725792598352138]
We propose a methodology for forecasting-temporal data using supervised semi-nonnegative matrix factorization (SMF) with frequency regularization.
We find that the results with the proposed methodology are comparable to previous research in the field of geophysical sciences but offer clearer interpretability.
arXiv Detail & Related papers (2023-11-15T01:23:13Z) - Transform Once: Efficient Operator Learning in Frequency Domain [69.74509540521397]
We study deep neural networks designed to harness the structure in frequency domain for efficient learning of long-range correlations in space or time.
This work introduces a blueprint for frequency domain learning through a single transform: transform once (T1)
arXiv Detail & Related papers (2022-11-26T01:56:05Z) - Conformal Frequency Estimation using Discrete Sketched Data with
Coverage for Distinct Queries [35.67445122503686]
This paper develops conformal inference methods to construct a confidence interval for the frequency of a queried object in a very large discrete data set.
We show our methods have improved empirical performance compared to existing frequentist and Bayesian alternatives in simulations.
arXiv Detail & Related papers (2022-11-09T00:05:29Z) - Conformalized Frequency Estimation from Sketched Data [6.510507449705344]
A flexible conformal inference method is developed to construct confidence intervals for the frequencies of objects queried in a very large data set.
The approach is completely data-adaptive and makes no use of any knowledge of the population distribution or of the inner workings of the sketching algorithm.
arXiv Detail & Related papers (2022-04-08T19:39:37Z) - Frequency Spectrum Augmentation Consistency for Domain Adaptive Object
Detection [107.52026281057343]
We introduce a Frequency Spectrum Augmentation Consistency (FSAC) framework with four different low-frequency filter operations.
In the first stage, we utilize all the original and augmented source data to train an object detector.
In the second stage, augmented source and target data with pseudo labels are adopted to perform the self-training for prediction consistency.
arXiv Detail & Related papers (2021-12-16T04:07:01Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.