State Diversity Matters in Offline Behavior Distillation

• URL: http://arxiv.org/abs/2512.06692v1
• Date: Sun, 07 Dec 2025 06:55:08 GMT
• Title: State Diversity Matters in Offline Behavior Distillation
• Abstract summary: We show that datasets with greater state diversity outperforms those with higher state quality when training loss is substantial.<n>We propose a novel yet simple algorithm, state density weighted (SDW) OBD, which emphasizes state diversity by weighting the distillation objective.
• Score: 58.116300485427764
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Offline Behavior Distillation (OBD), which condenses massive offline RL data into a compact synthetic behavioral dataset, offers a promising approach for efficient policy training and can be applied across various downstream RL tasks. In this paper, we uncover a misalignment between original and distilled datasets, observing that a high-quality original dataset does not necessarily yield a superior synthetic dataset. Through an empirical analysis of policy performance under varying levels of training loss, we show that datasets with greater state diversity outperforms those with higher state quality when training loss is substantial, as is often the case in OBD, whereas the relationship reverses under minimal loss, which contributes to the misalignment. By associating state quality and diversity in reducing pivotal and surrounding error, respectively, our theoretical analysis establishes that surrounding error plays a more crucial role in policy performance when pivotal error is large, thereby highlighting the importance of state diversity in OBD scenario. Furthermore, we propose a novel yet simple algorithm, state density weighted (SDW) OBD, which emphasizes state diversity by weighting the distillation objective using the reciprocal of state density, thereby distilling a more diverse state information into synthetic data. Extensive experiments across multiple D4RL datasets confirm that SDW significantly enhances OBD performance when the original dataset exhibits limited state diversity.

Related papers

• Spectral Gradient Surgery for Domain-Generalizable Dataset Distillation [6.809496808804485]
  We introduce Domain Generalizable dataset Distillation (DGDD)<n>We study the problem through a widely adopted Distribution Matching (DM) baseline.<n>We propose Spectral Gradient Surgery (SGS) to disentangle the two.
  arXiv  Detail & Related papers  (2026-05-13T02:33:38Z)
• Beyond Dataset Distillation: Lossless Dataset Concentration via Diffusion-Assisted Distribution Alignment [43.678155518039745]
  We propose a diffusion-based Noise-Optimization (N) method to synthesize a small yet representative set of samples, and "Opt" to augment the synthetic data via "Doping"<n>DsCo is applicable in both data-accessible and data-free scenarios, achieving SOTA performances for low data volumes, and it extends well to high data volumes, where it nearly reduces the dataset size by half with no performance degradation.
  arXiv  Detail & Related papers  (2026-03-30T03:20:27Z)
• Diffusion Models as Dataset Distillation Priors [39.4727398182562]
  We propose As Diffusion Priors (DAP), which formalizes representativeness by quantifying the similarity between synthetic and real data in feature space.<n>DAP outperforms state-of-the-art methods in generating high-fidelity datasets.<n>Our work establishes a theoretical connection between diffusion priors and the objectives of dataset distillation.
  arXiv  Detail & Related papers  (2025-10-20T11:04:09Z)
• Beyond Real Data: Synthetic Data through the Lens of Regularization [9.459299281438074]
  Synthetic data can improve generalization when real data is scarce, but excessive reliance may introduce distributional mismatches that degrade performance.<n>We present a learning-theoretic framework to quantify the trade-off between synthetic and real data.
  arXiv  Detail & Related papers  (2025-10-09T11:33:09Z)
• An Advanced Convolutional Neural Network for Bearing Fault Diagnosis under Limited Data [5.351573093028336]
  We propose an advanced data augmentation and contrastive fourier convolution framework (DAC-FCF) for bearing fault diagnosis under limited data.<n>Experiments demonstrate that DAC-FCF achieves significant improvements, outperforming baselines by up to 32%.
  arXiv  Detail & Related papers  (2025-09-14T02:41:48Z)
• How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradients [24.699470675718466]
  We present a spectral analysis of layer-wise gradients induced by low/high-quality instruction and reasoning data.<n>Our work illuminates the interplay between data quality and training stability, shedding novel insights into developing better data exploration strategies for post-training.
  arXiv  Detail & Related papers  (2025-04-14T23:53:47Z)
• Conditional Data Synthesis Augmentation [4.3108820946281945]
  Conditional Data Synthesis Augmentation (CoDSA) is a novel framework that synthesizes high-fidelity data for improving model performance across multimodal domains.<n>CoDSA fine-tunes pre-trained generative models to enhance the realism of synthetic data and increase sample density in sparse areas.<n>We introduce a theoretical framework that quantifies the statistical accuracy improvements enabled by CoDSA as a function of synthetic sample volume and targeted region allocation.
  arXiv  Detail & Related papers  (2025-04-10T03:38:11Z)
• Synthetic Data is an Elegant GIFT for Continual Vision-Language Models [52.343627275005026]
  GIFT is a novel continual fine-tuning approach to overcome catastrophic forgetting in Vision-Language Models.<n>We employ a pre-trained diffusion model to recreate both pre-training and learned downstream task data.<n>Our method consistently outperforms previous state-of-the-art approaches across various settings.
  arXiv  Detail & Related papers  (2025-03-06T09:09:18Z)
• Offline Behavior Distillation [57.6900189406964]
  Massive reinforcement learning (RL) data are typically collected to train policies offline without the need for interactions. We formulate offline behavior distillation (OBD), which synthesizes limited expert behavioral data from sub-optimal RL data. We propose two naive OBD objectives, DBC and PBC, which measure distillation performance via the decision difference between policies trained on distilled data and either offline data or a near-expert policy.
  arXiv  Detail & Related papers  (2024-10-30T06:28:09Z)
• Importance-Aware Adaptive Dataset Distillation [53.79746115426363]
  Development of deep learning models is enabled by the availability of large-scale datasets. dataset distillation aims to synthesize a compact dataset that retains the essential information from the large original dataset. We propose an importance-aware adaptive dataset distillation (IADD) method that can improve distillation performance.
  arXiv  Detail & Related papers  (2024-01-29T03:29:39Z)
• CAFE: Learning to Condense Dataset by Aligning Features [72.99394941348757]
  We propose a novel scheme to Condense dataset by Aligning FEatures (CAFE) At the heart of our approach is an effective strategy to align features from the real and synthetic data across various scales. We validate the proposed CAFE across various datasets, and demonstrate that it generally outperforms the state of the art.
  arXiv  Detail & Related papers  (2022-03-03T05:58:49Z)
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data [135.64775986546505]
  We study how to incorporate the dataset (observational data) collected offline, which is often abundantly available in practice, to improve the sample efficiency in the online setting. We propose the deconfounded optimistic value iteration (DOVI) algorithm, which incorporates the confounded observational data in a provably efficient manner.
  arXiv  Detail & Related papers  (2020-06-22T14:49:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.