A Small-Scale System for Autoregressive Program Synthesis Enabling Controlled Experimentation

• URL: http://arxiv.org/abs/2602.09112v1
• Date: Mon, 09 Feb 2026 19:03:04 GMT
• Title: A Small-Scale System for Autoregressive Program Synthesis Enabling Controlled Experimentation
• Abstract summary: We present a system called Cadmus which includes an integer virtual machine (VM), a dataset composed of true programs of diverse tasks, and an autoregressive transformer model that is trained for under $200 of compute cost.<n>The system can be used to study program completion, out-of-distribution representations, inductive reasoning, and instruction following in a setting where researchers have effective and affordable fine-grained control of the training distribution.
• Score: 4.499832265081113
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: What research can be pursued with small models trained to complete true programs? Typically, researchers study program synthesis via large language models (LLMs) which introduce issues such as knowing what is in or out of distribution, understanding fine-tuning effects, understanding the effects of tokenization, and higher demand on compute and storage to carry out experiments. We present a system called Cadmus which includes an integer virtual machine (VM), a dataset composed of true programs of diverse tasks, and an autoregressive transformer model that is trained for under \$200 of compute cost. The system can be used to study program completion, out-of-distribution representations, inductive reasoning, and instruction following in a setting where researchers have effective and affordable fine-grained control of the training distribution and the ability to inspect and instrument models. Smaller models working on complex reasoning tasks enable instrumentation and investigations that may be prohibitively expensive on larger models. To demonstrate that these tasks are complex enough to be of interest, we show that these Cadmus models outperform GPT-5 (by achieving 100\% accuracy while GPT-5 has 95\% accuracy) even on a simple task of completing correct, integer arithmetic programs in our domain-specific language (DSL) while providing transparency into the dataset's relationship to the problem. We also show that GPT-5 brings unknown priors into its reasoning process when solving the same tasks, demonstrating a confounding factor that prevents the use of large-scale LLMs for some investigations where the training set relationship to the task needs to be fully understood.

Related papers

• What do Language Models Learn and When? The Implicit Curriculum Hypothesis [50.89698323917259]
  We propose the Implicit Curriculum Hypothesis: pretraining follows a compositional and predictable curriculum across models.<n>We track emergence points across four model families spanning sizes from 410M-13B parameters.<n>We find that emergence orderings of when models reach fixed accuracy thresholds are strikingly consistent.
  arXiv  Detail & Related papers  (2026-04-09T17:50:12Z)
• SPaRFT: Self-Paced Reinforcement Fine-Tuning for Large Language Models [51.74498855100541]
  Large language models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL)<n>We propose textbfSPaRFT, a self-paced learning framework that enables efficient learning based on the capability of the model being trained.
  arXiv  Detail & Related papers  (2025-08-07T03:50:48Z)
• Case Studies of Generative Machine Learning Models for Dynamical Systems [0.5499796332553707]
  This paper focuses on two case studies of optimally controlled systems that are commonly understood and employed in aircraft guidance.<n>We report GAIMs that are trained with a relatively small set, of the order of a few hundred, of examples and with underlying governing equations.<n>New models are able to synthesize data that satisfy the governing equations and are statistically similar to the training data despite small volumes of training data.
  arXiv  Detail & Related papers  (2025-08-06T13:59:14Z)
• Mathematical Reasoning for Unmanned Aerial Vehicles: A RAG-Based Approach for Complex Arithmetic Reasoning [14.652057895402006]
  Autonomous UAV operation requires reliable mathematical reasoning for tasks such as trajectory planning and power management.<n>We propose RAG-UAV, a retrieval-augmented generation framework designed to improve the mathematical reasoning of several Large Language Models.<n>Our experiments demonstrate that incorporating retrieval substantially increases exact answer accuracy (achieving up to 75% with o1), reduces instances of incorrect formulation selection, and decreases numerical errors.
  arXiv  Detail & Related papers  (2025-06-05T13:09:24Z)
• Improving Small-Scale Large Language Models Function Calling for Reasoning Tasks [0.8425561594225592]
  This study introduces a novel framework for training smaller language models in function calling. It focuses on specific logical and mathematical reasoning tasks. The approach aims to improve performances of small-scale models for these tasks using function calling.
  arXiv  Detail & Related papers  (2024-10-24T16:27:35Z)
• Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewards [49.7719149179179]
  This paper investigates the feasibility of using PPO for reinforcement learning (RL) from explicitly programmed reward signals. We focus on tasks expressed through formal languages, such as programming, where explicit reward functions can be programmed to automatically assess quality of generated outputs. Our results show that pure RL-based training for the two formal language tasks is challenging, with success being limited even for the simple arithmetic task.
  arXiv  Detail & Related papers  (2024-10-22T15:59:58Z)
• Interpreting and Improving Large Language Models in Arithmetic Calculation [72.19753146621429]
  Large language models (LLMs) have demonstrated remarkable potential across numerous applications. In this work, we delve into uncovering a specific mechanism by which LLMs execute calculations. We investigate the potential benefits of selectively fine-tuning these essential heads/MLPs to boost the LLMs' computational performance.
  arXiv  Detail & Related papers  (2024-09-03T07:01:46Z)
• Scalable Learning of Item Response Theory Models [48.91265296134559]
  Item Response Theory (IRT) models aim to assess latent abilities of $n$ examinees along with latent difficulty characteristics of $m$ test items from categorical data. We leverage the similarity of these models to logistic regression, which can be approximated accurately using small weighted subsets called coresets.
  arXiv  Detail & Related papers  (2024-03-01T17:12:53Z)
• DeepCode AI Fix: Fixing Security Vulnerabilities with Large Language Models [3.1690235522182104]
  Large language models (LLMs) are increasingly used to solve various programming tasks. We show that the task is difficult as it requires the model to learn long-range code relationships. We propose a technique to address these challenges with a new approach for querying and fine-tuning LLMs.
  arXiv  Detail & Related papers  (2024-02-19T18:35:40Z)
• Limits of Transformer Language Models on Learning to Compose Algorithms [77.2443883991608]
  We evaluate training LLaMA models and prompting GPT-4 and Gemini on four tasks demanding to learn a composition of several discrete sub-tasks. Our results indicate that compositional learning in state-of-the-art Transformer language models is highly sample inefficient.
  arXiv  Detail & Related papers  (2024-02-08T16:23:29Z)
• Efficient Prompting via Dynamic In-Context Learning [76.83516913735072]
  We propose DynaICL, a recipe for efficient prompting with black-box generalist models. DynaICL dynamically allocates in-context examples according to the input complexity and the computational budget. We find that DynaICL saves up to 46% token budget compared to the common practice that allocates the same number of in-context examples to each input.
  arXiv  Detail & Related papers  (2023-05-18T17:58:31Z)
• CodeGen2: Lessons for Training LLMs on Programming and Natural Languages [116.74407069443895]
  We unify encoder and decoder-based models into a single prefix-LM. For learning methods, we explore the claim of a "free lunch" hypothesis. For data distributions, the effect of a mixture distribution and multi-epoch training of programming and natural languages on model performance is explored.
  arXiv  Detail & Related papers  (2023-05-03T17:55:25Z)
• A Survey of Learning on Small Data: Generalization, Optimization, and Challenge [101.27154181792567]
  Learning on small data that approximates the generalization ability of big data is one of the ultimate purposes of AI. This survey follows the active sampling theory under a PAC framework to analyze the generalization error and label complexity of learning on small data. Multiple data applications that may benefit from efficient small data representation are surveyed.
  arXiv  Detail & Related papers  (2022-07-29T02:34:19Z)
• A Step Towards Efficient Evaluation of Complex Perception Tasks in Simulation [5.4954641673299145]
  We propose an approach that enables efficient large-scale testing using simplified low-fidelity simulators. Our approach relies on designing an efficient surrogate model corresponding to the compute intensive components of the task under test. We demonstrate the efficacy of our methodology by evaluating the performance of an autonomous driving task in the Carla simulator with reduced computational expense.
  arXiv  Detail & Related papers  (2021-09-28T13:50:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.