GRIP: Geometric Refinement and Adaptive Information Potential for Data Efficiency

• URL: http://arxiv.org/abs/2603.00031v1
• Date: Wed, 04 Feb 2026 15:23:24 GMT
• Title: GRIP: Geometric Refinement and Adaptive Information Potential for Data Efficiency
• Abstract summary: textbfGRIP (Geometric Refinement and Adaptive Information Potential) is a framework that unifies these dimensions by modeling the corpus as an information-dense geometric space.<n>Our work establishes a robust geometric foundation for adaptive data curation in large-scale pre-training.
• Score: 28.832806758251607
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The performance of Large Language Models (LLMs) is increasingly governed by data efficiency rather than raw scaling volume. However, existing selection methods often decouple global distribution balancing from local instance selection, compromising the hierarchical integrity of the training set. We introduce \textbf{GRIP} (Geometric Refinement and Adaptive Information Potential), a framework that unifies these dimensions by modeling the corpus as an information-dense geometric space. GRIP employs a \textbf{Rapid Adaptation Probe (RAP)} to quantify the information potential of semantic clusters, dynamically re-allocating the sampling budget to regions with the highest representation deficits. Subsequently, we perform Intra-Cluster Selection using a \textbf{length-rectified geometric prior} to counteract embedding density artifacts and preserve long-tail logical sequences. Extensive evaluations on Mixture-of-Experts (MoE) models up to 300B tokens demonstrate that GRIP consistently outperforms state-of-the-art baselines, \textbf{surpassing the performance of models trained on $3\times$ larger uncurated datasets}. Our work establishes a robust geometric foundation for adaptive data curation in large-scale pre-training.

Related papers

• CuBAS: Information Geometric Curvature-Based Adaptive Sampling for Supervised Classification [52.452902154360565]
  CuBAS is an information-geometric framework for adaptive data selection in supervised classification.<n>A labeled dataset can be viewed as a statistical manifold, on which local curvature, estimated via the ratio of second to first-order observed Fisher information, faithfully encodes the geometric complexity of the data distribution.
  arXiv  Detail & Related papers  (2026-07-03T09:36:02Z)
• Geometry-Aware Dataset Condensation for Diffusion Model Training [103.45641113998839]
  We propose to reformulate real subset selection as a geometry-aware distribution alignment problem.<n>By incorporating one-sided partial optimal transport, our method selectively aligns a compact subset with the full data distribution.<n>Experiments across diffusion variants, subset sizes, image resolutions, and training rounds show that our method achieves superior fidelity and distributional coverage.
  arXiv  Detail & Related papers  (2026-06-04T08:53:58Z)
• SEMASIA: A Large-Scale Dataset of Semantically Structured Latent Representations [12.185380843937196]
  We introduce SEMASIA, a large-scale collection of latent representations extracted from approximately 1,700 pretrained vision models.<n>We analyze the conceptual organization of individual latent spaces, showing consistent prototype-like clustering.<n>We perform a large-scale regression analysis of how pretraining-data complexity, specialization, transfer learning, augmentation, and model scale relate to geometric and probing properties of embeddings.
  arXiv  Detail & Related papers  (2026-05-10T11:42:36Z)
• Tracing the Roots: A Multi-Agent Framework for Uncovering Data Lineage in Post-Training LLMs [81.43302841109349]
  Post-training data plays a pivotal role in shaping the capabilities of Large Language Models (LLMs)<n>We propose an automated multi-agent framework to reconstruct the evolutionary graph of dataset development.
  arXiv  Detail & Related papers  (2026-04-12T06:24:07Z)
• Persistence-Augmented Neural Networks [2.7309692684728613]
  We propose a persistence-based data augmentation framework that encodes local gradient flow regions and their hierarchical evolution.<n>This representation is compatible with both convolutional and graph neural networks.<n>We show that pruning the base level of the hierarchy reduces memory usage while maintaining competitive performance.
  arXiv  Detail & Related papers  (2026-04-09T17:06:06Z)
• To Memorize or to Retrieve: Scaling Laws for RAG-Considerate Pretraining [24.61808957290675]
  We study the trade-off between pretraining corpus size and retrieval store size across a wide range of model and data scales.<n>We find that retrieval consistently improves performance over parametric-only baselines across model scales.
  arXiv  Detail & Related papers  (2026-04-01T10:26:03Z)
• Optimizing the Landscape of LLM Embeddings with Dynamic Exploratory Graph Analysis for Generative Psychometrics: A Monte Carlo Study [0.0]
  This study reframes embeddings as searchable landscapes and adapts Dynamic Exploratory Graph Analysis (DynEGA) to traverse embedding coordinates.<n>A large-scale Monte Carlo simulation embedded items representing five dimensions of grandiose narcissism using OpenAI's text-embedding-3-small model.<n>Results reveal that Total Entropy Fit Index (TEFI) and Normalized Mutual Information (NMI) leads to competing optimization trajectories across the embedding landscape.
  arXiv  Detail & Related papers  (2026-01-14T14:40:14Z)
• Scalable Graph Generative Modeling via Substructure Sequences [50.32639806800683]
  We introduce Generative Graph Pattern Machine (G$2$PM), a generative Transformer pre-training framework for graphs.<n>G$2$PM represents graph instances (nodes, edges, or entire graphs) as sequences of substructures.<n>It employs generative pre-training over the sequences to learn generalizable and transferable representations.
  arXiv  Detail & Related papers  (2025-05-22T02:16:34Z)
• Exploring Training and Inference Scaling Laws in Generative Retrieval [50.82554729023865]
  Generative retrieval reformulates retrieval as an autoregressive generation task, where large language models generate target documents directly from a query.<n>We systematically investigate training and inference scaling laws in generative retrieval, exploring how model size, training data scale, and inference-time compute jointly influence performance.
  arXiv  Detail & Related papers  (2025-03-24T17:59:03Z)
• Gaussian Ensemble Belief Propagation for Efficient Inference in High-Dimensional Systems [3.6773638205393198]
  Efficient inference in high-dimensional models is a central challenge in machine learning.<n>We introduce the Gaussian Ensemble Belief Propagation (GEnBP) algorithm.<n>We show that GEnBP outperforms existing belief methods in terms of accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2024-02-13T03:31:36Z)
• RGM: A Robust Generalizable Matching Model [49.60975442871967]
  We propose a deep model for sparse and dense matching, termed RGM (Robust Generalist Matching) To narrow the gap between synthetic training samples and real-world scenarios, we build a new, large-scale dataset with sparse correspondence ground truth. We are able to mix up various dense and sparse matching datasets, significantly improving the training diversity.
  arXiv  Detail & Related papers  (2023-10-18T07:30:08Z)
• SIGMA: Scale-Invariant Global Sparse Shape Matching [50.385414715675076]
  We propose a novel mixed-integer programming (MIP) formulation for generating precise sparse correspondences for non-rigid shapes. We show state-of-the-art results for sparse non-rigid matching on several challenging 3D datasets.
  arXiv  Detail & Related papers  (2023-08-16T14:25:30Z)
• T1: Scaling Diffusion Probabilistic Fields to High-Resolution on Unified Visual Modalities [69.16656086708291]
  Diffusion Probabilistic Field (DPF) models the distribution of continuous functions defined over metric spaces. We propose a new model comprising of a view-wise sampling algorithm to focus on local structure learning. The model can be scaled to generate high-resolution data while unifying multiple modalities.
  arXiv  Detail & Related papers  (2023-05-24T03:32:03Z)
• Nearest Neighborhood-Based Deep Clustering for Source Data-absent Unsupervised Domain Adaptation [33.394228127643494]
  In the classic setting of unsupervised domain adaptation (UDA), the labeled source data are available in the training phase. In many real-world scenarios, the source data is inaccessible, and only a model trained on the source domain is available. This paper proposes a novel deep clustering method for this challenging task.
  arXiv  Detail & Related papers  (2021-07-27T04:13:59Z)
• Mix Dimension in Poincar\'{e} Geometry for 3D Skeleton-based Action Recognition [57.98278794950759]
  Graph Convolutional Networks (GCNs) have already demonstrated their powerful ability to model the irregular data. We present a novel spatial-temporal GCN architecture which is defined via the Poincar'e geometry. We evaluate our method on two current largest scale 3D datasets.
  arXiv  Detail & Related papers  (2020-07-30T18:23:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.