Pre-Training on Software Engineering Texts: Effects on Domain Adaptation and General-Language Understanding

• URL: http://arxiv.org/abs/2607.06613v1
• Date: Tue, 07 Jul 2026 08:05:05 GMT
• Title: Pre-Training on Software Engineering Texts: Effects on Domain Adaptation and General-Language Understanding
• Abstract summary: Generalist and code-focused Language Models (LMs) are increasingly applied to software engineering (SE)<n>We study how to adapt encoder and decoder LMs to SE text, comparing continual pre-training (CPT) against pre-training from scratch (PTS) on a new SE corpus.<n>We find that across families and sizes, reusing an existing LM dominates training a domain-native one from scratch.
• Score: 6.35250194032193
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generalist and code-focused Language Models (LMs) are increasingly applied to software engineering (SE), yet whether they are optimized for understanding SE textual artifacts (e.g., issues, commit messages, developer discussions) remains unclear, as most evidence comes from code-focused benchmarks. We study how to adapt encoder and decoder LMs to SE text, comparing continual pre-training (CPT) against pre-training from scratch (PTS) on a new SE corpus, and evaluating both domain adaptation (SELU) and general-language understanding (SuperGLUE). To keep the comparisons fair, we control pre-training under constant-token and compute-matched budgets. We find that across families and sizes, reusing an existing LM dominates training a domain-native one from scratch: CPT yields small and mostly inconclusive domain gains while leaving general-language understanding essentially unchanged, whereas PTS pays a large and usually decisive penalty on both axes and becomes competitive only for small LMs under a token-rich budget. We distill these results into practical guidance for adapting LMs to SE text and release our corpus and pre-trained LMs in our replication kit.

Related papers

• SEDCoT: Enhancing LLM-Based COBOL Code Translation via Symbolic Execution and Delta Debugging [49.898541267264115]
  Maintenance is increasingly challenging due to outdated technologies, sparse documentation, and developer retirement.<n>Traditional rule-based transcompilers yield outputs that are difficult to read and maintain.<n>We propose SEDCoT, a novel-to-C translation framework.
  arXiv  Detail & Related papers  (2026-07-05T03:07:10Z)
• What Really Improves Mathematical Reasoning: Structured Reasoning Signals Beyond Pure Code [72.9921566968371]
  We revisit the claim that code improves reasoning through controlled pretraining experiments on a 10T-token corpus with fine-grained domain separation.<n>Code substantially improves programming ability but does not act as a general reasoning enhancer.<n>Cross-domain structured reasoning traces, such as code-text and math-text mixtures, are better explained.
  arXiv  Detail & Related papers  (2026-05-19T12:37:01Z)
• LLM Reasoning for Machine Translation: Synthetic Data Generation over Thinking Tokens [25.257363122413395]
  "Thinking tokens" do not help LRMs better perform machine translation.<n>Fine-tuning a model with synthetic CoT explanations does not outperform standard input-output fine-tuning.<n>Our results suggest that using a teacher to refine target translations or to expand parallel corpora is more impactful than distilling their CoT explanations into "thinking" MT models.
  arXiv  Detail & Related papers  (2025-10-13T20:41:01Z)
• Redefining Machine Simultaneous Interpretation: From Incremental Translation to Human-Like Strategies [4.487634497356904]
  Simultaneous Machine Translation (SiMT) requires high-quality translations under strict real-time constraints.<n>We extend the action space of SiMT with four adaptive actions: SENTENCE_CUT, DROP, PARTIAL_MARIZATION and PRONOMINALIZATION.<n>We implement these actions in a decoder-only large language model (LLM) framework and construct training references through action-aware prompting.
  arXiv  Detail & Related papers  (2025-09-26T02:57:36Z)
• Context-aware Prompt Tuning: Advancing In-Context Learning with Adversarial Methods [69.36397993451742]
  This work introduces Context-aware Prompt Tuning (CPT), a method inspired by ICL, PT, and adversarial attacks. We modify specific context tokens, considering the unique structure of input and output formats. Inspired by adversarial attacks, we adjust the input based on the labels present in the context, focusing on minimizing, rather than maximizing, the loss.
  arXiv  Detail & Related papers  (2024-10-22T17:45:47Z)
• LeTI: Learning to Generate from Textual Interactions [60.425769582343506]
  We explore LMs' potential to learn from textual interactions (LETI) that not only check their correctness with binary labels but also pinpoint and explain errors in their outputs through textual feedback. Our focus is the code generation task, where the model produces code based on natural language instructions. LETI iteratively fine-tunes the model, using the objective LM, on a concatenation of natural language instructions, LM-generated programs, and textual feedback.
  arXiv  Detail & Related papers  (2023-05-17T15:53:31Z)
• An Interpretability Evaluation Benchmark for Pre-trained Language Models [37.16893581395874]
  We propose a novel evaluation benchmark providing with both English and Chinese annotated data. It tests LMs abilities in multiple dimensions, i.e., grammar, semantics, knowledge, reasoning and computation. It contains perturbed instances for each original instance, so as to use the rationale consistency under perturbations as the metric for faithfulness.
  arXiv  Detail & Related papers  (2022-07-28T08:28:09Z)
• Pre-training Text-to-Text Transformers for Concept-centric Common Sense [48.11844351407072]
  We propose a concept-aware language model (CALM) to augment pre-trained language models with concept-centric commonsense knowledge. We show that CALM can pack more commonsense knowledge into the parameters of a pre-trained text-to-text transformer without relying on external knowledge graphs.
  arXiv  Detail & Related papers  (2020-10-24T07:00:37Z)
• Byte Pair Encoding is Suboptimal for Language Model Pretraining [49.30780227162387]
  We analyze differences between unigram LM tokenization and byte-pair encoding (BPE) We find that the unigram LM tokenization method matches or outperforms BPE across downstream tasks and two languages. We hope that developers of future pretrained LMs will consider adopting the unigram LM method over the more prevalent BPE.
  arXiv  Detail & Related papers  (2020-04-07T21:21:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.