Minibatch Selection via Partition Matroid Constrained Gradient Matching

• URL: http://arxiv.org/abs/2606.07954v1
• Date: Sat, 06 Jun 2026 03:16:19 GMT
• Title: Minibatch Selection via Partition Matroid Constrained Gradient Matching
• Abstract summary: Training large language models (LLMs) on heterogeneous data requires selecting minibatches that balance convergence speed with coverage across domains.<n>We propose PartitionSel, a cross-domain minibatch selection approach that maximizes a validation-guided gradient-matching utility under per-domain budgets encoded as a partition-matroid constraint.
• Score: 19.541112219411556
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training large language models (LLMs) on heterogeneous data requires selecting minibatches that balance convergence speed with coverage across domains. Existing methods either select samples independently within each domain or rely on computationally expensive proxy models to learn continuous domain weights. We propose PartitionSel, a cross-domain minibatch selection approach that maximizes a validation-guided gradient-matching utility under per-domain budgets encoded as a partition-matroid constraint. By coupling the per-domain budgets through a single utility, PartitionSel is designed to reduce redundancy in selections across domains. The proposed objective is weakly submodular and admits an orthogonal matching pursuit algorithm with provable approximation guarantees. Empirically, we evaluate PartitionSel for minibatch selection during the fine-tuning of Qwen2.5 and Llama-3 on MetaMathQA and Mol-Instructions. PartitionSel achieves robust gains over per-domain and domain-agnostic baselines on both benchmarks. It also reduces the number of conflicting gradient pairs within each batch, indicating that the cross-domain coupling translates into more compatible training updates.

Related papers

• Clustering-Based Validation Splits for Model Selection under Domain Shift [0.0]
  It is proposed that the training-validation split should maximise the distribution mismatch between the two sets.<n>A constrained clustering algorithm, which leverages linear programming to control the size, label, and (optionally) group of the splits, is presented.
  arXiv  Detail & Related papers  (2024-05-29T19:21:17Z)
• Fast One-Stage Unsupervised Domain Adaptive Person Search [17.164485293539833]
  Unsupervised person search aims to localize a particular target person from a gallery set of scene images without annotations. We propose a Fast One-stage Unsupervised person Search (FOUS) which integrates complementary domain adaptaion with label adaptaion. FOUS can achieve the state-of-the-art (SOTA) performance on two benchmark datasets, CUHK-SYSU and PRW.
  arXiv  Detail & Related papers  (2024-05-05T07:15:47Z)
• Compositional Semantic Mix for Domain Adaptation in Point Cloud Segmentation [65.78246406460305]
  compositional semantic mixing represents the first unsupervised domain adaptation technique for point cloud segmentation. We present a two-branch symmetric network architecture capable of concurrently processing point clouds from a source domain (e.g. synthetic) and point clouds from a target domain (e.g. real-world)
  arXiv  Detail & Related papers  (2023-08-28T14:43:36Z)
• Bidirectional Domain Mixup for Domain Adaptive Semantic Segmentation [73.3083304858763]
  This paper systematically studies the impact of mixup under the domain adaptaive semantic segmentation task. In specific, we achieve domain mixup in two-step: cut and paste. We provide extensive ablation experiments to empirically verify our main components of the framework.
  arXiv  Detail & Related papers  (2023-03-17T05:22:44Z)
• Semi-supervised Domain Adaptive Structure Learning [72.01544419893628]
  Semi-supervised domain adaptation (SSDA) is a challenging problem requiring methods to overcome both 1) overfitting towards poorly annotated data and 2) distribution shift across domains. We introduce an adaptive structure learning method to regularize the cooperation of SSL and DA.
  arXiv  Detail & Related papers  (2021-12-12T06:11:16Z)
• Style Mixing and Patchwise Prototypical Matching for One-Shot Unsupervised Domain Adaptive Semantic Segmentation [21.01132797297286]
  In one-shot unsupervised domain adaptation, segmentors only see one unlabeled target image during training. We propose a new OSUDA method that can effectively relieve such computational burden. Our method achieves new state-of-the-art performance on two commonly used benchmarks for domain adaptive semantic segmentation.
  arXiv  Detail & Related papers  (2021-12-09T02:47:46Z)
• Minibatch vs Local SGD with Shuffling: Tight Convergence Bounds and Beyond [63.59034509960994]
  We study shuffling-based variants: minibatch and local Random Reshuffling, which draw gradients without replacement. For smooth functions satisfying the Polyak-Lojasiewicz condition, we obtain convergence bounds which show that these shuffling-based variants converge faster than their with-replacement counterparts. We propose an algorithmic modification called synchronized shuffling that leads to convergence rates faster than our lower bounds in near-homogeneous settings.
  arXiv  Detail & Related papers  (2021-10-20T02:25:25Z)
• Cross-Domain Grouping and Alignment for Domain Adaptive Semantic Segmentation [74.3349233035632]
  Existing techniques to adapt semantic segmentation networks across the source and target domains within deep convolutional neural networks (CNNs) do not consider an inter-class variation within the target domain itself or estimated category. We introduce a learnable clustering module, and a novel domain adaptation framework called cross-domain grouping and alignment. Our method consistently boosts the adaptation performance in semantic segmentation, outperforming the state-of-the-arts on various domain adaptation settings.
  arXiv  Detail & Related papers  (2020-12-15T11:36:21Z)
• Coded Stochastic ADMM for Decentralized Consensus Optimization with Edge Computing [113.52575069030192]
  Big data, including applications with high security requirements, are often collected and stored on multiple heterogeneous devices, such as mobile devices, drones and vehicles. Due to the limitations of communication costs and security requirements, it is of paramount importance to extract information in a decentralized manner instead of aggregating data to a fusion center. We consider the problem of learning model parameters in a multi-agent system with data locally processed via distributed edge nodes. A class of mini-batch alternating direction method of multipliers (ADMM) algorithms is explored to develop the distributed learning model.
  arXiv  Detail & Related papers  (2020-10-02T10:41:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.