Can Safety Emerge from Weak Supervision? A Systematic Analysis of Small Language Models

• URL: http://arxiv.org/abs/2603.07017v1
• Date: Sat, 07 Mar 2026 03:29:53 GMT
• Title: Can Safety Emerge from Weak Supervision? A Systematic Analysis of Small Language Models
• Abstract summary: Self-MOA is a fully automated framework for aligning small language models using weak supervision from automated evaluator models.<n>It achieves a 12.41% improvement in safety while preserving helpfulness, using as little as 11 times less training data than human-supervised alignment baselines.
• Score: 6.576811280895214
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Safety alignment is critical for deploying large language models (LLMs) in real-world applications, yet most existing approaches rely on large human-annotated datasets and static red-teaming benchmarks that are costly, difficult to scale, and slow to adapt to evolving model behaviors. Moreover, overly conservative safety mechanisms can reduce model usefulness by rejecting sensitive but legitimate queries. We introduce Self-MOA (Self Multi-Objective Alignment), a fully automated framework for aligning small language models using weak supervision from automated evaluator models. Self-MOA operates as a closed loop that dynamically generates model-specific red team prompts, constructs preference data from model-generated responses, and aligns models via multi-objective preference optimization to jointly optimize for safety and helpfulness. Across multiple small language models and safety benchmarks, Self-MOA achieves a 12.41\% improvement in safety while preserving helpfulness, using as little as 11 times less training data than human-supervised alignment baselines. These results demonstrate that adaptive, automated alignment can reduce the dependence on static, human-curated safety pipelines in resource-constrained settings.

Related papers

• CoMerge: Conflict-Driven Preference Optimization for Multi-Task Model Merging [59.836482609414816]
  We propose a conflict-driven preference optimization framework for model merging (CoMerge)<n>CoMerge enables the model to mitigate parameter-space conflicts while preserving task-specific capabilities.<n>Experiments show that CoMerge achieves an average normalized performance of 0.9968 on MergeBench.
  arXiv  Detail & Related papers  (2026-09-02T08:21:28Z)
• Generating Attacks for LLMs with GFlowNets [1.5749416770494706]
  This research aims to generate more effective adversarial attacks in English compared to existing benchmarks.<n>As a novel contribution to the literature, introduces a model capable of generating attack inputs in the Turkish language.
  arXiv  Detail & Related papers  (2026-08-10T19:39:10Z)
• Can Recommender Systems Teach Themselves? A Recursive Self-Improving Framework with Fidelity Control [82.30868101940068]
  We propose a paradigm in which a model bootstraps its own performance without reliance on external data or teacher models.<n>Our theoretical analysis shows that RSIR acts as a data-driven implicit regularizer, smoothing the optimization landscape.<n>We show that even smaller models benefit, and weak models can generate effective training curricula for stronger ones.
  arXiv  Detail & Related papers  (2026-02-17T15:31:32Z)
• Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflection [71.8243083897721]
  Vision-language models often hallucinate details, generating non-existent objects or inaccurate attributes that compromise output reliability.<n>We present a novel framework that leverages the model's self-consistency between long responses and short answers to generate preference pairs for training.
  arXiv  Detail & Related papers  (2025-09-27T10:37:11Z)
• SALMAN: Stability Analysis of Language Models Through the Maps Between Graph-based Manifolds [11.373585987937913]
  We propose a unified, local (sample-level) robustness framework (SALMAN) that evaluates model stability without modifying internal parameters or resorting to complex perturbations.<n>Central to our approach is a novel Distance Mapping Distortion (DMD) measure, which ranks each sample's susceptibility by comparing input-to-output distance mappings in a near-linear manner.<n>By demonstrating significant gains in attack efficiency and robust training, we position our framework as a practical, model-agnostic tool for advancing the reliability of transformer-based NLP systems.
  arXiv  Detail & Related papers  (2025-08-23T02:50:55Z)
• SAFER: Probing Safety in Reward Models with Sparse Autoencoder [13.390179692415233]
  We present sparse Autoencoder For Enhanced Reward model (textbfSAFER)<n>We uncover human-interpretable features in reward model activations, enabling insight into safety-relevant decision-making.<n>Experiments show that SAFER can precisely degrade or enhance safety alignment with minimal data modification.
  arXiv  Detail & Related papers  (2025-07-01T11:04:03Z)
• From Failures to Fixes: LLM-Driven Scenario Repair for Self-Evolving Autonomous Driving [29.36624509719055]
  We propose textbfSERA, a framework that enables autonomous driving systems to self-evolve by repairing failure cases through targeted scenario recommendation.<n>By analyzing performance logs, SERA identifies failure patterns and dynamically retrieves semantically aligned scenarios from a structured bank.<n>Experiments on the benchmark show that SERA consistently improves key metrics across multiple autonomous driving baselines, demonstrating its effectiveness and generalizability under safety-critical conditions.
  arXiv  Detail & Related papers  (2025-05-28T07:46:19Z)
• Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judge [1.1666234644810893]
  Small models outperform larger ones in safety, suggesting that training and architecture may matter more than scale.<n>No model is fully robust to adversarial elicitation, with jailbreak attacks using low-resource languages or refusal suppression proving effective.
  arXiv  Detail & Related papers  (2025-04-10T16:00:59Z)
• More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment [80.04449725137177]
  Direct Preference Optimization (DPO) has emerged as a simple, yet effective alternative to reinforcement learning from human feedback.<n>Our study reveals a striking, safety-specific phenomenon associated with DPO alignment.<n>Using solely self-generated responses for both chosen and rejected pairs significantly outperforms configurations that incorporate responses from stronger models.
  arXiv  Detail & Related papers  (2025-04-03T00:36:40Z)
• SafeRoute: Adaptive Model Selection for Efficient and Accurate Safety Guardrails in Large Language Models [63.63254955809224]
  We propose a binary router that distinguishes hard examples from easy ones.<n>Our method selectively applies the larger safety guard model to the data that the router considers hard, improving efficiency while maintaining accuracy.<n> Experimental results on multiple benchmark datasets demonstrate that our adaptive model selection significantly enhances the trade-off between computational cost and safety performance.
  arXiv  Detail & Related papers  (2025-02-18T02:51:17Z)
• Towards Stable Machine Learning Model Retraining via Slowly Varying Sequences [6.067007470552307]
  We propose a model-agnostic framework for finding sequences of models that are stable across retraining iterations.<n>We develop a mixed-integer optimization formulation that is guaranteed to recover optimal models.<n>We find that, on average, a 2% reduction in predictive power leads to a 30% improvement in stability.
  arXiv  Detail & Related papers  (2024-03-28T22:45:38Z)
• AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving [68.73885845181242]
  We propose an Automatic Data Engine (AIDE) that automatically identifies issues, efficiently curates data, improves the model through auto-labeling, and verifies the model through generation of diverse scenarios. We further establish a benchmark for open-world detection on AV datasets to comprehensively evaluate various learning paradigms, demonstrating our method's superior performance at a reduced cost.
  arXiv  Detail & Related papers  (2024-03-26T04:27:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.