Accelerated Dynamic Importance Weighting with Versatile Divergence-Minimizing Estimators

• URL: http://arxiv.org/abs/2605.25499v1
• Date: Mon, 25 May 2026 06:58:51 GMT
• Title: Accelerated Dynamic Importance Weighting with Versatile Divergence-Minimizing Estimators
• Abstract summary: We propose accelerated DIW (ADIW), a unified and efficient IW framework for deep learning under joint distribution shift.<n>ADIW generalizes DIW into a unified divergence-minimization framework that supports diverse weight-estimation methods.<n>ADIW achieves state-of-the-art IW performance while being substantially more efficient.
• Score: 62.20244820580353
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Importance weighting (IW) is a golden solver for joint distribution shift, where the joint distributions differ between the training and test data. To solve this problem, IW estimates test-to-training density ratios as importance weights and reweights the training losses accordingly. Recent advances in dynamic IW (DIW) integrate weight estimation into model training, enabling scalable IW for deep models and achieving strong performance on large modern datasets. Despite its promise, DIW remains limited in two aspects. First, it incurs substantial computational overhead by solving a kernel mean matching (KMM)-induced optimization problem to convergence in every mini-batch. Second, it relies solely on KMM for weight estimation, whereas the IW literature contains diverse estimation methods based on different divergence measures. In this paper, we propose accelerated DIW (ADIW), a unified and efficient IW framework for deep learning under joint distribution shift. ADIW performs a few lightweight projected gradient descent updates that warm-start from previously updated weights, substantially improving efficiency. Moreover, ADIW generalizes DIW into a unified divergence-minimization framework that supports diverse weight-estimation methods in a plug-and-play manner, including those based on the Kullback-Leibler divergence, squared distance, and Wasserstein-1 distance. We establish convergence guarantees for ADIW under mild conditions, and empirical results demonstrate that ADIW achieves state-of-the-art IW performance while being substantially more efficient.

Related papers

• SAW: Stage-Aware Dynamic Weighting for Multi-Objective Reinforcement Learning in Large Language Models [66.92079232778707]
  We propose Stage-Aware Dynamic Weighting (SAW) as a lightweight, algorithm-agnostic dynamic weighting mechanism.<n>SAW reweights each dimension's reward or advantage contribution by its relative informativeness within the batch.<n> Experiments on tool-calling and text summarization tasks demonstrate that SAW consistently improves both training efficiency and final performance.
  arXiv  Detail & Related papers  (2026-06-05T10:00:19Z)
• DyWeight: Dynamic Gradient Weighting for Few-Step Diffusion Sampling [18.524470249911232]
  Dynamic Gradient Weighting (DyWeight) is a lightweight, learning-based multi-step solver.<n>DyWeight learns unconstrained time-varying parameters that adaptively aggregate historical gradients.<n>Experiments on CIFAR-10, FFHQ, AFHQv2, ImageNet64, LSUN-Bedroom, Stable Diffusion and FLUX.1-dev demonstrate that DyWeight achieves superior visual fidelity and stability.
  arXiv  Detail & Related papers  (2026-03-12T07:01:44Z)
• MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources [113.33902847941941]
  Variance-Aware Sampling (VAS) is a data selection strategy guided by Variance Promotion Score (VPS)<n>We release large-scale, carefully curated resources containing 1.6M long CoT cold-start data and 15k RL QA pairs.<n> Experiments across mathematical reasoning benchmarks demonstrate the effectiveness of both the curated data and the proposed VAS.
  arXiv  Detail & Related papers  (2025-09-25T14:58:29Z)
• AHDMIL: Asymmetric Hierarchical Distillation Multi-Instance Learning for Fast and Accurate Whole-Slide Image Classification [51.525891360380285]
  AHDMIL is an Asymmetric Hierarchical Distillation Multi-Instance Learning framework.<n>It eliminates irrelevant patches through a two-step training process.<n>It consistently outperforms previous state-of-the-art methods in both classification performance and inference speed.
  arXiv  Detail & Related papers  (2025-08-07T07:47:16Z)
• A General Adaptive Dual-level Weighting Mechanism for Remote Sensing Pansharpening [11.791358860917189]
  deep learning methods for remote sensing pansharpening have advanced rapidly.<n>Many existing methods struggle to fully leverage feature heterogeneity and redundancy.<n>We introduce a general adaptive dual-level weighting mechanism (ADWM) to address these challenges.
  arXiv  Detail & Related papers  (2025-03-17T14:24:00Z)
• MARS: Unleashing the Power of Variance Reduction for Training Large Models [56.67982828148859]
  We propose a unified training framework for deep neural networks.<n>We introduce three instances of MARS that leverage preconditioned gradient optimization.<n>Results indicate that the implementation of MARS consistently outperforms Adam.
  arXiv  Detail & Related papers  (2024-11-15T18:57:39Z)
• Weight Scope Alignment: A Frustratingly Easy Method for Model Merging [40.080926444789085]
  Non-I.I.D. data poses a huge challenge for averaging-based model fusion. In this paper, we reveal variations in weight scope under different training conditions, shedding light on its influence on model merging. Fortunately, the parameters in each layer basically follow the Gaussian distribution, which inspires a novel and simple regularization approach.
  arXiv  Detail & Related papers  (2024-08-22T09:13:27Z)
• Diverse Weight Averaging for Out-of-Distribution Generalization [100.22155775568761]
  We propose Diverse Weight Averaging (DiWA) to average weights obtained from several independent training runs rather than from a single run. DiWA consistently improves the state of the art on the competitive DomainBed benchmark without inference overhead.
  arXiv  Detail & Related papers  (2022-05-19T17:44:22Z)
• Rethinking Importance Weighting for Deep Learning under Distribution Shift [86.52964129830706]
  Under distribution shift (DS) where the training data distribution differs from the test one, a powerful technique is importance weighting (IW) which handles DS in two separate steps. In this paper, we rethink IW and theoretically show it suffers from a circular dependency. We propose an end-to-end solution dynamic IW that iterates between WE and WC and combines them in a seamless manner.
  arXiv  Detail & Related papers  (2020-06-08T15:03:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.