SpecAlign: Efficient Specification-Grounded Alignment of Large Language Models via Synthetic Data

• URL: http://arxiv.org/abs/2606.16276v2
• Date: Wed, 17 Jun 2026 06:16:12 GMT
• Title: SpecAlign: Efficient Specification-Grounded Alignment of Large Language Models via Synthetic Data
• Abstract summary: We propose specification-grounded alignment, a new alignment paradigm that treats provider-authored model specifications as the primary alignment target.<n>We introduce SpecAlign, a framework that synthesizes alignment data directly from specification documents.
• Score: 46.83043333985018
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As large language models (LLMs) are increasingly deployed in real-world applications, alignment is no longer governed by a single universal notion of safety or helpfulness, but instead by provider- or application-specific model specifications. These specifications are typically long, structured, and frequently updated, yet existing alignment pipelines lack a systematic mechanism to operationalize them as training signals. In this paper, we propose specification-grounded alignment, a new alignment paradigm that treats provider-authored model specifications as the primary alignment target rather than abstract principles or static benchmarks. To instantiate this paradigm, we introduce SpecAlign, a framework that synthesizes alignment data directly from specification documents. SpecAlign combines structured rule annotation, controllable specification instantiation, and multi-agent adversarial data synthesis to generate fine-grained, boundary-aware preference pairs that capture both compliant behaviors and meaningful specification violations. Experiments across multiple model specifications and backbone models demonstrate that training with SpecAlign consistently improves rule compliance while preserving general capabilities and avoiding over-conservative behavior. These results suggest that grounding alignment in explicit model specifications enables rapid, precise, and scalable adaptation of LLM behavior to evolving policy requirements.

Related papers

• Formal Verification of Agentic Systems over Operational Data [59.98246281888422]
  Agentic systems driven by large language models (LLMs) are increasingly deployed in real-world where they act on persistent operational data.<n>Existing approaches do not provide such system-level guarantees.<n>We study here the verification of agentic systems comprising a single LLM and a tool orchestration harness over relational operational data.
  arXiv  Detail & Related papers  (2026-08-04T13:01:30Z)
• Position: It's Time to Optimize LLMs for Self-Consistency [50.68587056599734]
  Many important failures persist in language model pre- and post-training pipelines.<n>We argue that these failures arise from a modeling assumption permeating all aspects of the pipeline.<n>We propose self-consistency as a framework for understanding these failures.
  arXiv  Detail & Related papers  (2026-07-31T19:16:53Z)
• Towards Spec Learning: Inference-Time Alignment from Preference Pairs [0.10195618602298682]
  We propose spec learning, a framework that relies on a brief user instruction and a small set of preference judgments.<n>We show that the responses generated based on the compiled specifications often outperform direct preference optimization (DPO) on datasets from specialized domains.<n>Unlike weight updates, the resulting specifications are human-readable and double as interpretable and transparent written embodiments of the preference signal that produced them.
  arXiv  Detail & Related papers  (2026-06-22T23:21:55Z)
• Neuro-Symbolic Agents for Hallucination-Free Requirements Reuse [0.0]
  We present a neuro-symbolic multi-agent system that re-conceptualizes requirements reuse as a Model-Driven Elicitation process.<n>Our system achieves 100% requirement coverage and a constraint-violation rate of only 0.2%.
  arXiv  Detail & Related papers  (2026-05-02T18:16:04Z)
• VeriAct: Beyond Verifiability -- Agentic Synthesis of Correct and Complete Formal Specifications [0.3240750198587795]
  We compare classical and prompt-based approaches for automated JML specification synthesis.<n>We investigate whether prompt optimization can push synthesis quality further by evolving prompts through structured verification feedback.<n>We propose VeriAct, a verification-guided agentic framework that iteratively synthesizes and repairs specifications.
  arXiv  Detail & Related papers  (2026-03-31T22:12:15Z)
• Specification-Driven Generation and Evaluation of Discrete-Event World Models via the DEVS Formalism [9.578714623643291]
  We seek a principled middle ground that combines the reliability of explicit simulators with the flexibility of learned models.<n>We advocate explicit, executable discrete-event world models synthesized directly from natural-language specifications.
  arXiv  Detail & Related papers  (2026-03-04T06:50:32Z)
• Alignment-Aware Model Adaptation via Feedback-Guided Optimization [27.93864970404945]
  Fine-tuning is the primary mechanism for adapting foundation models to downstream tasks.<n>We propose an alignment-aware fine-tuning framework that integrates feedback from an external alignment signal through policy-gradient-based regularization.
  arXiv  Detail & Related papers  (2026-02-02T16:03:16Z)
• Data Dependency-Aware Code Generation from Enhanced UML Sequence Diagrams [54.528185120850274]
  We propose a novel step-by-step code generation framework named API2Dep.<n>First, we introduce an enhanced Unified Modeling Language (UML) API diagram tailored for service-oriented architectures.<n>Second, recognizing the critical role of data flow, we introduce a dedicated data dependency inference task.
  arXiv  Detail & Related papers  (2025-08-05T12:28:23Z)
• Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models [48.15777554876988]
  Traditional alignment methods often require retraining large pretrained models.<n>We propose a novel textitResidual Alignment Model (textitRAM) that formalizes the alignment process as a type of importance sampling.<n>We develop a resampling algorithm with iterative token-level decoding to address the common first-token latency issue in comparable methods.
  arXiv  Detail & Related papers  (2025-05-26T08:53:02Z)
• Spatial-Temporal-Spectral Unified Modeling for Remote Sensing Dense Prediction [20.1863553357121]
  Current deep learning architectures for remote sensing are fundamentally rigid.<n>We introduce the Spatial-Temporal-Spectral Unified Network (STSUN) for unified modeling.<n> STSUN can adapt to input and output data with arbitrary spatial sizes, temporal lengths, and spectral bands.<n>It unifies various dense prediction tasks and diverse semantic class predictions.
  arXiv  Detail & Related papers  (2025-05-18T07:39:17Z)
• SoFA: Shielded On-the-fly Alignment via Priority Rule Following [90.32819418613407]
  This paper introduces a novel alignment paradigm, priority rule following, which defines rules as the primary control mechanism in each dialog. We present PriorityDistill, a semi-automated approach for distilling priority following signals from simulations to ensure robust rule integration and adherence.
  arXiv  Detail & Related papers  (2024-02-27T09:52:27Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• Target-Embedding Autoencoders for Supervised Representation Learning [111.07204912245841]
  This paper analyzes a framework for improving generalization in a purely supervised setting, where the target space is high-dimensional. We motivate and formalize the general framework of target-embedding autoencoders (TEA) for supervised prediction, learning intermediate latent representations jointly optimized to be both predictable from features as well as predictive of targets.
  arXiv  Detail & Related papers  (2020-01-23T02:37:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.