MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection

• URL: http://arxiv.org/abs/2605.30288v2
• Date: Fri, 29 May 2026 06:16:52 GMT
• Title: MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection
• Abstract summary: We propose MIRA, a source-aware filtering framework based on self-anchored rubric discovery.<n>On code-oriented mid-training with 21 sources and 5 source groups, MIRA outperforms selection baselines across nine code benchmarks.
• Score: 59.34281845159177
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Mid-training has become an important stage in modern LLM development, using large-scale curated mixtures to strengthen capabilities before final post-training. Its data selection problem is distinct: the data are optimized under a pretraining-style objective at near-pretraining scale, but are curated toward downstream capabilities and drawn from heterogeneous sources with different formats and training roles. As a result, effective selection requires both scalability and source-adaptive semantic criteria. Existing model-based methods scale well, but provide only implicit quality signals. Semantic selection methods offer stronger judgments, but usually assume fixed rubrics or standardized data formats. To address this mismatch, we propose MIRA, a source-aware filtering framework based on self-anchored rubric discovery. The key idea is to make rubric construction part of data selection: MIRA first discovers what should be evaluated for each source group, then distills those judgments into scalable student scorers for full-corpus filtering. On code-oriented mid-training with 21 sources and 5 source groups, MIRA outperforms selection baselines across nine code benchmarks and matches the full-corpus run while using only half the tokens.

Related papers

• Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation [6.262888209373497]
  We investigate how language models encode preference information in their intermediate representations.<n>We find that activations from chosen and rejected responses form distinct clusters across layers, even in pretrained models.<n>Exploiting this structure, we propose training a lightweight linear probe on a few labelled preference pairs and using it to annotate large unlabelled datasets.
  arXiv  Detail & Related papers  (2026-08-31T14:52:19Z)
• Diversity-Based Active Learning: An Evaluation of Metric Spaces for Active Learning Selection [0.0]
  Pool-based active learning minimizes costs by querying only the most informative samples from an unlabeled dataset.<n> Diversity-based approaches, on the other hand, attempt to select a representative subset of the data.<n>We will focus on evaluating the performance of Greedy K-center across a variety of metric spaces.
  arXiv  Detail & Related papers  (2026-08-24T16:30:10Z)
• OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training [14.608696839111134]
  We propose OptiMer, which decouples ratio selection from training.<n>Experiments on Gemma 3 27B across languages (Japanese, Chinese) and domains (Math, Code)
  arXiv  Detail & Related papers  (2026-03-30T18:00:02Z)
• When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets [29.94723846950853]
  We present the first comprehensive, data-centric analysis of popular open-source DPO corpora.<n>We leverage the Magpie framework to annotate each sample for task category, input quality, and preference reward.<n>This enables a scalable, fine-grained inspection of preference quality across datasets, revealing structural and qualitative discrepancies in reward margins.
  arXiv  Detail & Related papers  (2025-11-14T06:12:16Z)
• Composable Cross-prompt Essay Scoring by Merging Models [7.5702468122067685]
  Cross-prompt automated essay scoring typically trains models jointly on all source prompts.<n>We propose a source-free adaptation approach that selectively merges individually trained source models' parameters instead of datasets.
  arXiv  Detail & Related papers  (2025-05-24T06:28:21Z)
• Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinement [8.509688686402438]
  Finetuning large language models on instruction data is crucial for enhancing pre-trained knowledge and improving instruction-following capabilities. This work addresses the question: How can we determine the optimal subset of data for effective training? Our method employs k-means clustering to ensure the selected subset effectively represents the full dataset.
  arXiv  Detail & Related papers  (2024-09-17T17:25:31Z)
• How to Train Data-Efficient LLMs [56.41105687693619]
  We study data-efficient approaches for pre-training language models (LLMs) We find that Ask-LLM and Density sampling are the best methods in their respective categories. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories.
  arXiv  Detail & Related papers  (2024-02-15T02:27:57Z)
• Data Selection for Language Models via Importance Resampling [90.9263039747723]
  We formalize the problem of selecting a subset of a large raw unlabeled dataset to match a desired target distribution. We extend the classic importance resampling approach used in low-dimensions for LM data selection. We instantiate the DSIR framework with hashed n-gram features for efficiency, enabling the selection of 100M documents in 4.5 hours.
  arXiv  Detail & Related papers  (2023-02-06T23:57:56Z)
• Divide and Contrast: Source-free Domain Adaptation via Adaptive Contrastive Learning [122.62311703151215]
  Divide and Contrast (DaC) aims to connect the good ends of both worlds while bypassing their limitations. DaC divides the target data into source-like and target-specific samples, where either group of samples is treated with tailored goals. We further align the source-like domain with the target-specific samples using a memory bank-based Maximum Mean Discrepancy (MMD) loss to reduce the distribution mismatch.
  arXiv  Detail & Related papers  (2022-11-12T09:21:49Z)
• BMD: A General Class-balanced Multicentric Dynamic Prototype Strategy for Source-free Domain Adaptation [74.93176783541332]
  Source-free Domain Adaptation (SFDA) aims to adapt a pre-trained source model to the unlabeled target domain without accessing the well-labeled source data. To make up for the absence of source data, most existing methods introduced feature prototype based pseudo-labeling strategies. We propose a general class-Balanced Multicentric Dynamic prototype strategy for the SFDA task.
  arXiv  Detail & Related papers  (2022-04-06T13:23:02Z)
• Improving Contrastive Learning on Imbalanced Seed Data via Open-World Sampling [96.8742582581744]
  We present an open-world unlabeled data sampling framework called Model-Aware K-center (MAK) MAK follows three simple principles: tailness, proximity, and diversity. We demonstrate that MAK can consistently improve both the overall representation quality and the class balancedness of the learned features.
  arXiv  Detail & Related papers  (2021-11-01T15:09:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.