Active Learning with Low-Rank Structure for Data Selection
Abstract Overview
This paper studies batch data selection through a low-rank view of dataset structure rather than a clustering view. It proposes a sensitivity-sampling framework that uses low-rank sketches, residuals, and weighted sampling to build a small subset whose weighted average loss approximates the full-dataset average loss under stated regularity assumptions. The theory gives coreset-style guarantees with subset size scaling as approximately \(\tilde{O}(k + 1/\varepsilon^2)\) in the abstract, and the main theorems show high-probability loss approximation up to additive terms involving the best rank-\(k\) approximation cost. Empirically, the method is evaluated on a credit-card classification dataset and on LLM fine-tuning tasks using Llama3-8B and Qwen2.5-3B.
Novelty
The distinctive contribution is a data-selection framework based on low-rank approximation and row subset selection, aimed at preserving dominant spectral structure instead of relying on geometric clustering objectives such as k-center or k-means. The work also connects this low-rank sampling strategy to explicit loss-approximation guarantees for weighted coresets.
Results
In the credit-card experiments, the low-rank sensitivity method achieves the lowest coreset approximation error across tested sample sizes and yields higher downstream logistic-regression accuracy than random sampling and clustering-based selection. In Llama3-8B fine-tuning, the method consistently outperforms uniform sampling and is broadly competitive with clustering-based sensitivity sampling across GSM8k, ViGGO, and SQL. On Qwen2.5-3B for ViGGO, it exceeds all reported baselines at both 25% and 12.5% compression, and subsets selected with Llama3-8B transfer effectively to Qwen2.5-3B.
Key Points
- The method samples data according to low-rank sensitivity scores derived from projections onto a selected subspace and residual norms, then assigns importance weights to form a coreset.
- Theoretical guarantees show that the weighted subset can approximate the dataset’s total or average loss within relative error terms plus an additive component tied to the optimal rank-k approximation error.
- Experiments suggest that some datasets are better captured by low-rank structure than clustering structure, which helps explain the empirical gains over uniform and clustering-based baselines.