Data-Efficient Language Modeling: From Frontier Advancement to Principle-Guided Model Improvement

• URL: http://arxiv.org/abs/2609.10702v1
• Date: Wed, 09 Sep 2026 18:00:52 GMT
• Title: Data-Efficient Language Modeling: From Frontier Advancement to Principle-Guided Model Improvement
• Abstract summary: Qiushi conducted a long-horizon, end-to-end autonomous research program on BabyLM 2026 Strict-Small.<n>Three stages connected frontier advancement, principle discovery, and principle-guided model improvement.
• Score: 17.36807692618427
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning from limited text requires models to use context, generalize to new inputs, and retain useful capabilities. Qiushi Engine conducted a long-horizon, end-to-end autonomous research program on BabyLM 2026 Strict-Small, within 10 million corpus words and 100 million cumulative word presentations. Three stages connected frontier advancement, principle discovery, and principle-guided model improvement. Stage I combined compact restatements, budget reinvestment, and residual incremental learning to build a frontier model. Stage II found that exact repetition and aligned restatement produce different patterns of context use, depending on target relations and prediction windows. In controlled tasks, recovering familiar performance did not ensure that unseen inputs could still use learned computations. These findings support a testable data-efficient learning principle: organize experience around the contextual dependencies needed for prediction; separately design visible information, supervision, and preservation; test learning, generalization, and retention. Stage III retained source text, masked more local clues, supervised selected targets, and preserved predictions on ordinarily masked inputs. Two continuation seeds from the same parent outperformed ordinary continuation on the complete nine-metric aggregate. Overall rose from 42.02 to 42.25 across two generations; the second achieved the highest Overall in the public Strict-Small snapshot of 8 September 2026. Further studies addressed compression, relational anchors, shared representations, and measurement. Models are available on Hugging Face; code and research records accompany the GitHub repository. Together, these stages illustrate Research RSI: recursive self-improvement of the research process. Scientific understanding and method innovations change subsequent questions and designs; new experiments test and refine them.

Related papers

• A Dataset for Modeling Iterative Problem-Solving [26.98102675986399]
  We build a benchmark that evaluates models spanning parametric, sequential, and generative traditions.<n>We find that the model's coding proficiency is inversely related to predictive performance in this setting.
  arXiv  Detail & Related papers  (2026-09-01T09:00:55Z)
• Combining Trained Models in Reinforcement Learning [0.0]
  Deep reinforcement learning (DRL) has delivered strong results in domains such as Atari and Go, but it still suffers from high sample cost and weak transfer beyond the training setting.<n>A common response is to reuse information from previously trained models through transfer, distillation, ensemble methods, or federated training.<n>This paper presents a PRISMA-guided systematic review of empirical studies on pretrained knowledge reuse in DRL.
  arXiv  Detail & Related papers  (2026-05-04T02:37:55Z)
• PrefixNLI: Detecting Factual Inconsistencies as Soon as They Arise [60.63315470285562]
  MiniTruePrefixes is a novel specialized model that better detects factual inconsistencies over text prefixes.<n>We show that integrating MiniTruePrefixes into a controlled decoding framework substantially improves factual consistency in abstractive summarization.
  arXiv  Detail & Related papers  (2025-11-03T09:07:44Z)
• Detect Anything via Next Point Prediction [51.55967987350882]
  Rex- Omni is a 3B-scale MLLM that achieves state-of-the-art object perception performance.<n>On benchmarks like COCO and LVIS, Rex- Omni attains performance comparable to or exceeding regression-based models.
  arXiv  Detail & Related papers  (2025-10-14T17:59:54Z)
• Representation-Based Exploration for Language Models: From Test-Time to Post-Training [50.144031964319424]
  Reinforcement learning (RL) promises to expand the capabilities of language models.<n>It is unclear if current RL techniques promote the discovery of novel behaviors, or simply sharpen those already present in the base model.<n>We investigate the value of deliberate exploration -- explicitly incentivizing the model to discover novel and diverse behaviors.
  arXiv  Detail & Related papers  (2025-10-13T17:49:05Z)
• Large Language Models Imitate Logical Reasoning, but at what Cost? [0.42970700836450487]
  We present a study which evaluates the reasoning capability of frontier Large Language Models over an eighteen month period.<n>We measured the accuracy of three leading models from December 2023, September 2024 and June 2025 on true or false questions.<n>The improvement in performance from 2023 to 2024 can be attributed to hidden Chain of Thought prompting.
  arXiv  Detail & Related papers  (2025-09-16T04:03:42Z)
• Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language Models [81.62767292169225]
  We investigate knowledge forgetting in large language models with a focus on its generalisation.<n>We propose PerMU, a novel probability perturbation-based unlearning paradigm.<n>Experiments are conducted on a diverse range of datasets, including TOFU, Harry Potter, ZsRE, WMDP, and MUSE.
  arXiv  Detail & Related papers  (2025-02-27T11:03:33Z)
• The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute [66.84421705029624]
  We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours. We pre-process an existing large, diverse, and high-quality dataset of books that surpasses existing academic benchmarks in quality, diversity, and document length. This work also provides two baseline models: a feed-forward model derived from the GPT-2 architecture and a recurrent model in the form of a novel LSTM with ten-fold throughput.
  arXiv  Detail & Related papers  (2023-09-20T10:31:17Z)
• Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction [49.15931834209624]
  We present the first benchmark that simulates the evaluation of open information extraction models in the real world.<n>We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique.<n>By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques.
  arXiv  Detail & Related papers  (2023-05-23T12:05:09Z)
• NEVIS'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Research [96.53307645791179]
  We introduce the Never-Ending VIsual-classification Stream (NEVIS'22), a benchmark consisting of a stream of over 100 visual classification tasks. Despite being limited to classification, the resulting stream has a rich diversity of tasks from OCR, to texture analysis, scene recognition, and so forth. Overall, NEVIS'22 poses an unprecedented challenge for current sequential learning approaches due to the scale and diversity of tasks.
  arXiv  Detail & Related papers  (2022-11-15T18:57:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.