GRASP: Geometry-aware Residual Alignment for Scalable Pretraining Data Attribution

• URL: http://arxiv.org/abs/2606.06892v1
• Date: Fri, 05 Jun 2026 04:17:50 GMT
• Title: GRASP: Geometry-aware Residual Alignment for Scalable Pretraining Data Attribution
• Abstract summary: This work reframes attribution as subset-level counterfactual utility prediction.<n>We introduce GRASP, an interaction-aware surrogate.<n>It decisively outperforms existing scalable baselines.
• Score: 5.770893169582546
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scalable data attribution methods typically assign isolated utility scores to individual training examples. This prevalent additive assumption fundamentally fails to capture critical subset dynamics, including data redundancy and complementary coverage. In this work, we reframe attribution as subset-level counterfactual utility prediction and introduce GRASP, an interaction-aware surrogate. Grounded in a theoretical smoothness lower bound, GRASP explicitly models subset interactions through a quadratic geometric penalty. To achieve pretraining-scale efficiency without relying on hidden oracle tuning, we couple low-dimensional feature sketches with a strictly finite lower-confidence bound selection protocol. Extensive subset-retraining evaluations demonstrate that GRASP decisively outperforms existing scalable baselines. It more than doubles the task-level rank correlation for counterfactual subset fidelity while reducing upfront artifact construction costs by nearly an order of magnitude. Downstream diagnostics further show that this scoring mechanism transfers to language model curation and cross-domain vision selection, establishing a robust foundation for optimizing massive pretraining corpora.

Related papers

• An Ensemble-Based Self-Taught Learning Approach for Parking Space Classification Under Limited Data [0.0]
  We investigate a self-taught learning framework based on unsupervised representation learning with convolutional autoencoders.<n>The proposed approach learns transferable visual representations from unlabeled data and reuses the learned encoders as fixed feature extractors for supervised classification.<n> Experiments conducted on the PKLot and CNRPark benchmarks show that the proposed ensemble-based strategy substantially reduces annotation requirements.
  arXiv  Detail & Related papers  (2026-09-03T01:35:02Z)
• Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation [54.74045834035952]
  Vision-language dataset distillation (VLDD) compresses a large image-text paired dataset into a small set of synthetic pairs.<n>We propose a rank-aware hyperbolic alignment (RAHA) that combines hierarchical geometry with explicit alignment-capacity control.<n>RAHA demonstrates competitive cross-modal retrieval and improved transfer indicators under fixed budgets.
  arXiv  Detail & Related papers  (2026-06-28T15:41:31Z)
• UniRank: Unified Rank Allocation for Low-Rank LLM Compression [9.742379971053806]
  We formulate global low-rank allocation as a sorting-and-truncation pipeline.<n>We verify the strong correlation between high input-output cosine similarity and low effective rank.<n>We propose rank-preserving fine-tuning, which performs direct LoRA tuning on decomposed weights.
  arXiv  Detail & Related papers  (2026-06-20T03:10:33Z)
• Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models [65.89572755202245]
  Diffusion Language Models (DLMs) offer stronger global awareness and highly parallel generation.<n>Post-training DLMs with standard Negative Evidence Lower Bound (NELBO)-based supervised fine-tuning remains inefficient.<n>We propose a self-distilled trajectory-based post-training framework that aligns training with the easy-to-hard structure of inference.
  arXiv  Detail & Related papers  (2026-05-12T09:39:06Z)
• Evaluating the Efficiency of Latent Spaces via the Coupling-Matrix [0.5013248430919224]
  We introduce a redundancy index, denoted rho(C), that directly quantifies inter-dimensional dependencies.<n>Low rho(C) reliably predicts high classification accuracy or low reconstruction error, while elevated redundancy is associated with performance collapse.<n>We show that Tree-structured Parzen Estimators (TPE) preferentially explore low-rho regions, suggesting that rho(C) can guide neural architecture search and serve as a redundancy-aware regularization target.
  arXiv  Detail & Related papers  (2025-09-08T03:36:47Z)
• Robust and Label-Efficient Deep Waste Detection [29.019461511410515]
  Effective waste sorting is critical for sustainable recycling, yet AI research in this domain continues to lag behind commercial systems.<n>In this work, we advance AI-driven waste detection by establishing strong baselines and introducing an ensemble-based semi-supervised learning framework.
  arXiv  Detail & Related papers  (2025-08-26T08:34:04Z)
• Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing [58.52119063742121]
  Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance.<n>This paper addresses the question of how to optimally combine the model's predictions and the provided labels.<n>Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels.
  arXiv  Detail & Related papers  (2025-05-21T07:16:44Z)
• Optimizing Singular Spectrum for Large Language Model Compression [95.7621116637755]
  We introduce SoCo, a novel compression framework that learns to rescale the decomposed components of SVD in a data-driven manner.<n>Thanks to the learnable singular spectrum, SoCo adaptively prunes components according to the sparsified importance scores.<n> Experimental evaluations across multiple LLMs and benchmarks demonstrate that SoCo surpasses the state-of-the-art methods in model compression.
  arXiv  Detail & Related papers  (2025-02-20T23:18:39Z)
• Predict, Cluster, Refine: A Joint Embedding Predictive Self-Supervised Framework for Graph Representation Learning [0.0]
  Graph representation learning has emerged as a cornerstone for tasks like node classification and link prediction.<n>Current self-supervised learning (SSL) methods face challenges such as computational inefficiency, reliance on contrastive objectives, and representation collapse.<n>We propose a novel joint embedding predictive framework for graph SSL that eliminates contrastive objectives and negative sampling while preserving semantic and structural information.
  arXiv  Detail & Related papers  (2025-02-02T07:42:45Z)
• Consensus-Adaptive RANSAC [104.87576373187426]
  We propose a new RANSAC framework that learns to explore the parameter space by considering the residuals seen so far via a novel attention layer. The attention mechanism operates on a batch of point-to-model residuals, and updates a per-point estimation state to take into account the consensus found through a lightweight one-step transformer.
  arXiv  Detail & Related papers  (2023-07-26T08:25:46Z)
• Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression [53.15502562048627]
  Recent work has built the connection between self-supervised learning and the approximation of the top eigenspace of a graph Laplacian operator. This work delves into a statistical analysis of augmentation-based pretraining.
  arXiv  Detail & Related papers  (2023-06-01T15:18:55Z)
• PCPL: Predicate-Correlation Perception Learning for Unbiased Scene Graph Generation [58.98802062945709]
  We propose a novel Predicate-Correlation Perception Learning scheme to adaptively seek out appropriate loss weights. Our PCPL framework is further equipped with a graph encoder module to better extract context features.
  arXiv  Detail & Related papers  (2020-09-02T08:30:09Z)
• Target-Embedding Autoencoders for Supervised Representation Learning [111.07204912245841]
  This paper analyzes a framework for improving generalization in a purely supervised setting, where the target space is high-dimensional. We motivate and formalize the general framework of target-embedding autoencoders (TEA) for supervised prediction, learning intermediate latent representations jointly optimized to be both predictable from features as well as predictive of targets.
  arXiv  Detail & Related papers  (2020-01-23T02:37:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.