Understanding Dataset Distillation via Spectral Filtering

• URL: http://arxiv.org/abs/2503.01212v1
• Date: Mon, 03 Mar 2025 06:22:34 GMT
• Title: Understanding Dataset Distillation via Spectral Filtering
• Authors: Deyu Bo, Songhua Liu, Xinchao Wang,
• Abstract summary: We introduce UniDD, a spectral filtering framework that unifies diverse DD objectives.<n>UniDD reveals that the essence of DD fundamentally lies in matching frequency-specific features.<n>To address this limitation, we propose Curriculum Frequency Matching (CFM), which gradually adjusts the filter parameter to cover both low- and high-frequency information.
• Score: 69.07076441512612
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dataset distillation (DD) has emerged as a promising approach to compress datasets and speed up model training. However, the underlying connections among various DD methods remain largely unexplored. In this paper, we introduce UniDD, a spectral filtering framework that unifies diverse DD objectives. UniDD interprets each DD objective as a specific filter function that affects the eigenvalues of the feature-feature correlation (FFC) matrix and modulates the frequency components of the feature-label correlation (FLC) matrix. In this way, UniDD reveals that the essence of DD fundamentally lies in matching frequency-specific features. Moreover, according to the filter behaviors, we classify existing methods into low-frequency matching and high-frequency matching, encoding global texture and local details, respectively. However, existing methods rely on fixed filter functions throughout distillation, which cannot capture the low- and high-frequency information simultaneously. To address this limitation, we further propose Curriculum Frequency Matching (CFM), which gradually adjusts the filter parameter to cover both low- and high-frequency information of the FFC and FLC matrices. Extensive experiments on small-scale datasets, such as CIFAR-10/100, and large-scale datasets, including ImageNet-1K, demonstrate the superior performance of CFM over existing baselines and validate the practicality of UniDD.

Related papers

• Extra Clients at No Extra Cost: Overcome Data Heterogeneity in Federated Learning with Filter Decomposition [25.658632928800962]
  We propose a technique for decomposing a convolutional filter in federated learning (FL) into a linear combination of filter subspace elements. This simple technique transforms global filter aggregation in FL into aggregating filter atoms and their atom coefficients. Empirical results on benchmark datasets demonstrate that our filter decomposition technique substantially improves the accuracy of FL methods.
  arXiv  Detail & Related papers  (2025-03-11T17:42:36Z)
• Frequency-aware Feature Fusion for Dense Image Prediction [99.85757278772262]
  We propose Frequency-Aware Feature Fusion (FreqFusion) for dense image prediction tasks. FreqFusion integrates an Adaptive Low-Pass Filter (ALPF) generator, an offset generator, and an Adaptive High-Pass Filter (AHPF) generator. Comprehensive visualization and quantitative analysis demonstrate that FreqFusion effectively improves feature consistency and sharpens object boundaries.
  arXiv  Detail & Related papers  (2024-08-23T07:30:34Z)
• Frequency-Aware Deepfake Detection: Improving Generalizability through Frequency Space Learning [81.98675881423131]
  This research addresses the challenge of developing a universal deepfake detector that can effectively identify unseen deepfake images. Existing frequency-based paradigms have relied on frequency-level artifacts introduced during the up-sampling in GAN pipelines to detect forgeries. We introduce a novel frequency-aware approach called FreqNet, centered around frequency domain learning, specifically designed to enhance the generalizability of deepfake detectors.
  arXiv  Detail & Related papers  (2024-03-12T01:28:00Z)
• Frequency Domain-based Dataset Distillation [17.02955182740882]
  We present FreD, a novel parameterization method for dataset distillation. FreD employs frequency-based transforms to optimize the frequency representations of each data instance.
  arXiv  Detail & Related papers  (2023-11-15T09:46:30Z)
• Decomposed Guided Dynamic Filters for Efficient RGB-Guided Depth Completion [46.04264366475848]
  RGB-guided depth completion aims at predicting dense depth maps from sparse depth measurements and corresponding RGB images. Guided dynamic filters generate spatially-variant depth-wise separable convolutional filters from RGB features to guide depth features. We propose to decompose the guided dynamic filters into a spatially-shared component multiplied by content-adaptive adaptors at each spatial location.
  arXiv  Detail & Related papers  (2023-09-05T08:37:58Z)
• SpectralDiff: A Generative Framework for Hyperspectral Image Classification with Diffusion Models [18.391049303136715]
  We propose a generative framework for HSI classification with diffusion models (SpectralDiff) SpectralDiff effectively mines the distribution information of high-dimensional and highly redundant data. Experiments on three public HSI datasets demonstrate that the proposed method can achieve better performance than state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-04-12T16:32:34Z)
• Few-shot One-class Domain Adaptation Based on Frequency for Iris Presentation Attack Detection [33.41823375502942]
  Iris presentation attack detection (PAD) has achieved remarkable success to ensure the reliability and security of iris recognition systems. Most existing methods exploit discriminative features in the spatial domain and report outstanding performance under intra-dataset settings. We propose a new domain adaptation setting called Few-shot One-class Domain Adaptation (FODA), where adaptation only relies on a limited number of target bonafide samples.
  arXiv  Detail & Related papers  (2022-04-01T11:55:06Z)
• Deep Frequency Filtering for Domain Generalization [55.66498461438285]
  Deep Neural Networks (DNNs) have preferences for some frequency components in the learning process. We propose Deep Frequency Filtering (DFF) for learning domain-generalizable features. We show that applying our proposed DFF on a plain baseline outperforms the state-of-the-art methods on different domain generalization tasks.
  arXiv  Detail & Related papers  (2022-03-23T05:19:06Z)
• Hierarchical Dynamic Filtering Network for RGB-D Salient Object Detection [91.43066633305662]
  The main purpose of RGB-D salient object detection (SOD) is how to better integrate and utilize cross-modal fusion information. In this paper, we explore these issues from a new perspective. We implement a kind of more flexible and efficient multi-scale cross-modal feature processing.
  arXiv  Detail & Related papers  (2020-07-13T07:59:55Z)
• iffDetector: Inference-aware Feature Filtering for Object Detection [70.8678270164057]
  We introduce a generic Inference-aware Feature Filtering (IFF) module that can easily be combined with modern detectors. IFF performs closed-loop optimization by leveraging high-level semantics to enhance the convolutional features. IFF can be fused with CNN-based object detectors in a plug-and-play manner with negligible computational cost overhead.
  arXiv  Detail & Related papers  (2020-06-23T02:57:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.