How Post-Training Shapes Biological Reasoning Models

• URL: http://arxiv.org/abs/2606.16517v1
• Date: Mon, 15 Jun 2026 10:19:49 GMT
• Title: How Post-Training Shapes Biological Reasoning Models
• Abstract summary: We train and evaluate more than 100 biological reasoning models under controlled variation in backbone, continued pre-training, supervised fine-tuning, and reinforcement learning.<n>We find that each post-training stage reshapes generalization in a distinct way rather than contributing uniform gains.
• Score: 50.53183971442794
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scientific reasoning models for biology combine language models with foundation models trained on multimodal biological data, including DNA, RNA, and proteins. These models are built through post-training, yet how each stage shapes reasoning and generalization remains poorly understood. We study when post-training improves performance and when it induces over-specialization. Across genomics, transcriptomics, and proteins, we train and evaluate more than 100 biological reasoning models under controlled variation in backbone, continued pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning (RL), measuring both in-domain (ID) and out-of-domain (OOD) performance. We find that each post-training stage reshapes generalization in a distinct way rather than contributing uniform gains. CPT improves downstream performance by aligning models with biological language. SFT consistently increases ID performance but causes OOD performance to peak early and decline as models fit the training distribution. RL, when applied to strong SFT checkpoints with aligned rewards, improves OOD performance and partially recovers generalization. These results show that biological reasoning does not improve monotonically with additional supervision or compute. Instead, performance depends on how training stages are composed. Under fixed post-training budgets, the strongest ID-OOD trade-off comes from brief SFT, larger RL allocations, and asymmetric adaptation capacity across stages.

Related papers

• Reinforcement Learning Towards Broadly and Persistently Beneficial Models [5.737612356236512]
  We study whether RL on beneficial behavior, instantiated in realistic domains, can produce broad and persistent alignment generalization.<n>We construct a dataset of realistic situations designed to measure and train beneficial traits, including truthfulness, fairness, risk awareness, and corrigibility.<n>We then train models with RL on this dataset and evaluate them on more than 50 independent benchmarks of alignment and beneficial behavior.
  arXiv  Detail & Related papers  (2026-06-22T23:35:49Z)
• Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models [51.45122910254346]
  Reinforcement learning (RL)-based post-training often improves the reasoning performance of large language models beyond the training domain.<n>Supervised fine-tuning (SFT) frequently leads to general forgetting capabilities.<n>We present a feature-level mechanistic analysis methodology to probe RL generalization using a controlled experimental setup.
  arXiv  Detail & Related papers  (2026-04-27T21:22:34Z)
• Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Models [54.50728814348712]
  reinforcement learning (RL) has inspired the growing adoption of RL for post-training Multimodal Large Language Models (MLLMs) to enhance their visual reasoning capabilities.<n>We propose the Hallucination-as-Cue Framework to investigate the effects of RL-based post-training on multimodal reasoning models from the perspective of model hallucination.
  arXiv  Detail & Related papers  (2026-04-03T16:56:34Z)
• How and Why LLMs Generalize: A Fine-Grained Analysis of LLM Reasoning from Cognitive Behaviors to Low-Level Patterns [51.02752099869218]
  Large Language Models (LLMs) display strikingly different generalization behaviors.<n>We introduce a novel benchmark that decomposes reasoning into atomic core skills.<n>We show that RL-tuned models maintain more stable behavioral profiles and resist collapse in reasoning skills, whereas SFT models exhibit sharper drift and overfit to surface patterns.
  arXiv  Detail & Related papers  (2025-12-30T08:16:20Z)
• Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning [42.80470927369973]
  We study how model scale, data volume, and computational budget interact to shape performance.<n>We find that larger models trained for fewer steps consistently outperform smaller models trained for more steps.<n>In data-constrained regimes, repeated reuse of high-quality data proves highly effective.
  arXiv  Detail & Related papers  (2025-09-29T17:10:35Z)
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning [93.00629872970364]
  Reinforcement learning (RL) has become the dominant paradigm for improving the performance of language models on complex reasoning tasks.<n>We introduce SPARKLE, a fine-grained analytic framework to dissect the effects of RL across three key dimensions.<n>We study whether difficult problems -- those yielding no RL signals and mixed-quality reasoning traces -- can still be effectively used for training.
  arXiv  Detail & Related papers  (2025-06-05T07:53:59Z)
• Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining [74.83412846804977]
  Reinforcement learning (RL)-based fine-tuning has become a crucial step in post-training language models.<n>We present a systematic end-to-end study of RL fine-tuning for mathematical reasoning by training models entirely from scratch.
  arXiv  Detail & Related papers  (2025-04-10T17:15:53Z)
• Regularized Evolutionary Population-Based Training [11.624954122221562]
  This paper presents an algorithm called Population-Based Training (EPBT) that interleaves the training of a DNN's weights with the metalearning of loss functions. EPBT results in faster, more accurate learning on image classification benchmarks.
  arXiv  Detail & Related papers  (2020-02-11T06:28:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.