To Intervene or Not: Guiding Inference-time Alignment with Probabilistic Model Blending

• URL: http://arxiv.org/abs/2606.11201v1
• Date: Wed, 22 Apr 2026 05:57:29 GMT
• Title: To Intervene or Not: Guiding Inference-time Alignment with Probabilistic Model Blending
• Abstract summary: We introduce BlendIn, an inference-time alignment framework that shifts from binary decisions to creating hybrid distributions integrating both models' knowledge.<n> BlendIn stabilizes inference-time alignment by performing quality-aware alignment and proportionally weighting each model's contribution based on reliability.
• Score: 9.4417382684481
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The wide deployment of LLMs has made model alignment necessary to make newly trained models safely and effectively respond to user instructions. Among different methods, inference-time alignment is often cheaper as it intervenes (i.e., offers guidances) only during output generation. Existing proposals apply guidances extracted from certain aligned models without properly assessing their reliability. Nonetheless, our systematic evaluation reveals that guidance effectiveness varies drastically across models; since ineffective guidances lead to further confusion and thus further interventions, the resulting excessive interventions typically indicate poor performance. To make interventions more effective and thus more efficient, we introduce BlendIn, an inference-time alignment framework that shifts from binary decisions to creating hybrid distributions integrating both models' knowledge. BlendIn stabilizes inference-time alignment by performing quality-aware alignment and proportionally weighting each model's contribution based on reliability. Compared with existing works, it preserves beneficial guidance while downweighting unreliable suggestions. BlendIn provides both diagnostic signals and mitigation strategies for misaligned guidance, achieving consistent and up to 50% performance improvement on challenging model pairs. Our code is available at: https://github.com/DecayingSeart/BlendIn.

Related papers

• Inference-time Alignment via Sparse Junction Steering [25.464612964225484]
  Token-level steering has emerged as a pivotal approach for inference-time alignment.<n>Existing methods rely on dense intervention at every decoding step.<n>We show that dense intervention is unnecessary and propose sparse junction steering.
  arXiv  Detail & Related papers  (2026-01-30T08:40:47Z)
• Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging [35.958192369444056]
  "alignment tax" of post-training is typically framed as a drop in task accuracy.<n>We show it also involves a severe loss of calibration, making models overconfident, less reliable, and model outputs less diverse.<n>We show that this trade-off can be navigated effectively via a simple post-hoc intervention: interpolating between a model's weights before and after alignment.
  arXiv  Detail & Related papers  (2025-10-20T11:12:41Z)
• Steerable Adversarial Scenario Generation through Test-Time Preference Alignment [58.37104890690234]
  Adversarial scenario generation is a cost-effective approach for safety assessment of autonomous driving systems.<n>We introduce a new framework named textbfSteerable textbfAdversarial scenario textbfGEnerator (SAGE)<n>SAGE enables fine-grained test-time control over the trade-off between adversariality and realism without any retraining.
  arXiv  Detail & Related papers  (2025-09-24T13:27:35Z)
• DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models [50.32663816994459]
  Diffusion-styled Preference Optimization (model) provides an efficient and policy-agnostic solution for aligning LLMs with humans.<n>modelavoids the time latency associated with token-level generation.<n>Experiments on AlpacaEval 2, MT-bench, and HH-RLHF demonstrate that modelachieves superior alignment performance across various settings.
  arXiv  Detail & Related papers  (2025-03-06T09:21:54Z)
• Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback [64.67540769692074]
  Large language models (LLMs) fine-tuned with alignment techniques, such as reinforcement learning from human feedback, have been instrumental in developing some of the most capable AI systems to date.<n>We introduce an approach called Margin Matching Preference Optimization (MMPO), which incorporates relative quality margins into optimization, leading to improved LLM policies and reward models.<n>Experiments with both human and AI feedback data demonstrate that MMPO consistently outperforms baseline methods, often by a substantial margin, on popular benchmarks including MT-bench and RewardBench.
  arXiv  Detail & Related papers  (2024-10-04T04:56:11Z)
• LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks [52.46420522934253]
  We introduce LoRA-Ensemble, a parameter-efficient ensembling method for self-attention networks.<n>The method not only outperforms state-of-the-art implicit techniques like BatchEnsemble, but even matches or exceeds the accuracy of an Explicit Ensemble.
  arXiv  Detail & Related papers  (2024-05-23T11:10:32Z)
• InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance [56.184255657175335]
  We develop textbfInferAligner, a novel inference-time alignment method that utilizes cross-model guidance for harmlessness alignment. Experimental results show that our method can be very effectively applied to domain-specific models in finance, medicine, and mathematics. It significantly diminishes the Attack Success Rate (ASR) of both harmful instructions and jailbreak attacks, while maintaining almost unchanged performance in downstream tasks.
  arXiv  Detail & Related papers  (2024-01-20T10:41:03Z)
• Towards Calibrated Robust Fine-Tuning of Vision-Language Models [97.19901765814431]
  This work proposes a robust fine-tuning method that improves both OOD accuracy and confidence calibration simultaneously in vision language models. We show that both OOD classification and OOD calibration errors have a shared upper bound consisting of two terms of ID data. Based on this insight, we design a novel framework that conducts fine-tuning with a constrained multimodal contrastive loss enforcing a larger smallest singular value.
  arXiv  Detail & Related papers  (2023-11-03T05:41:25Z)
• Learning Under Adversarial and Interventional Shifts [36.183840774167756]
  We propose a new formulation, RISe, for designing robust models against a set of distribution shifts. We employ the distributionally robust optimization framework to optimize the resulting objective in both supervised and reinforcement learning settings.
  arXiv  Detail & Related papers  (2021-03-29T20:10:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.