Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignment

• URL: http://arxiv.org/abs/2605.13537v1
• Date: Wed, 13 May 2026 13:47:06 GMT
• Title: Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignment
• Abstract summary: Inference-time alignment techniques offer a lightweight alternative or complement to costly reinforcement learning.<n>We extend these techniques by introducing reference-model temperature adjustment.<n>To reward hacking, we propose an algorithm for calibrating SLOP weight parameters.
• Score: 20.603753314256704
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inference-time alignment techniques offer a lightweight alternative or complement to costly reinforcement learning, while enabling continual adaptation as alignment objectives and reward targets evolve. Existing theoretical analyses justify these methods as approximations to sampling from distributions optimally tilted toward a given reward model. We extend these techniques by introducing reference-model temperature adjustment, which leads to further generalization of inference-time alignment to ensembles of generative reward models combined as a sharpened logarithmic opinion pool (SLOP). To mitigate reward hacking, we propose an algorithm for calibrating SLOP weight parameters and experimentally demonstrate that it improves robustness while preserving alignment performance.

Related papers

• Advancing Optimal Subset Oracle via Learning Relaxation of Neural Set Functions [41.373161204333435]
  Learning neural set functions is pivotal to a wide range of important applications, including compound selection in AI-driven drug discovery and product recommendation.<n>Recent work has introduced optimal subsets to implicitly learn set functions under practical weakly supervised settings.<n>These frameworks rely on Monte Carlo sampling to estimate gradients of the evidence lower bound when updating the variational distribution.<n>In this work, we reinterpret the evidence lower bound as a continuous relaxation of the set function and learn a surrogate objective that replaces sampling-based ELBO estimation during variational optimization.
  arXiv  Detail & Related papers  (2026-07-13T13:37:47Z)
• Curvature-Guided LoRA: Steering in the pretrained NTK subspace [60.35296431630704]
  We introduce the prediction alignment problem, aiming to match the predictor obtained via PEFT to that of full fine-tuning at the level of outputs.<n>We show that this objective naturally leads to a curvature-aware, second-order formulation, where optimal low-rank updates correspond to a Newton-like, curvature-whitened gradient.<n>Based on this insight, we propose Curvature-Guided LoRA (CG-LoRA), which selects and scales adaptation directions using local curvature information.
  arXiv  Detail & Related papers  (2026-03-31T14:46:39Z)
• On the Role of Batch Size in Stochastic Conditional Gradient Methods [51.97510367368799]
  We study the role of batch size in conditional gradient methods under a $$-Kurdyka-ojasiewicz ($$-KL) condition.<n>Our study reveals a regime-dependent behavior: increasing the batch size initially improves optimization accuracy but, beyond a critical threshold, the benefits and saturate can degrade performance under a fixed token budget.
  arXiv  Detail & Related papers  (2026-03-22T12:23:41Z)
• Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignment [20.579613395916024]
  Inference-time alignment effectively steers large language models by generating multiple candidates from a reference model and selecting among them with an imperfect reward model.<n>We show that the optimal strategy depends critically on the tail behavior of the reward distribution.<n>We introduce Best-of-Tails (BoT), an adaptive inference-time alignment framework that uses Tsallis divergence as a tunable regularizer.
  arXiv  Detail & Related papers  (2026-03-06T19:00:54Z)
• Learnable Chernoff Baselines for Inference-Time Alignment [64.81256817158851]
  We introduce Learnable Chernoff Baselines as a method for efficiently and approximately sampling from exponentially tilted kernels.<n>We establish total-variation guarantees to the ideal aligned model, and demonstrate in both continuous and discrete diffusion settings that LCB sampling closely matches ideal rejection sampling.
  arXiv  Detail & Related papers  (2026-02-08T00:09:40Z)
• Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training [29.56905427210088]
  gradient-ARM is a framework that jointly optimize a rubric generator and a judge using reinforcement learning from preference feedback.<n>We show that gradient-ARM achieves state-of-the-art performance among baselines on benchmarks and significantly improves downstream policy alignment in both offline and online reinforcement learning settings.
  arXiv  Detail & Related papers  (2026-02-02T00:50:53Z)
• Taming Preference Mode Collapse via Directional Decoupling Alignment in Diffusion Reinforcement Learning [27.33241821967005]
  We propose a novel framework that mitigates Preference Mode Collapse (PMC)<n>D$2$-Align achieves superior alignment with human preference.
  arXiv  Detail & Related papers  (2025-12-30T11:17:52Z)
• Inference-Time Scaling of Diffusion Language Models with Particle Gibbs Sampling [70.8832906871441]
  We study how to steer generation toward desired rewards without retraining the models.<n>Prior methods typically resample or filter within a single denoising trajectory, optimizing rewards step-by-step without trajectory-level refinement.<n>We introduce particle Gibbs sampling for diffusion language models (PG-DLM), a novel inference-time algorithm enabling trajectory-level refinement while preserving generation perplexity.
  arXiv  Detail & Related papers  (2025-07-11T08:00:47Z)
• Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Models [26.211711150915203]
  $Psi$-Sampler is an SMC-based framework incorporating pCNL-based initial particle sampling.<n>Inference-time reward alignment with score-based generative models has gained significant traction.
  arXiv  Detail & Related papers  (2025-06-02T05:02:33Z)
• InfAlign: Inference-aware language model alignment [58.66389179049758]
  Language model alignment is a critical step in training modern generative language models.<n>We show that this train/test mismatch makes standard RLHF framework sub-optimal in view of inference-time methods.<n>We propose a framework for inference-aware alignment (InfAlign), which aims to optimize inference-time win rate of the aligned policy against the base model.
  arXiv  Detail & Related papers  (2024-12-27T18:45:36Z)
• Enhancing Fairness through Reweighting: A Path to Attain the Sufficiency Rule [23.335423207588466]
  We introduce an innovative approach to enhancing the empirical risk minimization process in model training. This scheme aims to uphold the sufficiency rule in fairness by ensuring that optimal predictors maintain consistency across diverse sub-groups.
  arXiv  Detail & Related papers  (2024-08-26T09:19:58Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)
• Robust Sampling in Deep Learning [62.997667081978825]
  Deep learning requires regularization mechanisms to reduce overfitting and improve generalization. We address this problem by a new regularization method based on distributional robust optimization. During the training, the selection of samples is done according to their accuracy in such a way that the worst performed samples are the ones that contribute the most in the optimization.
  arXiv  Detail & Related papers  (2020-06-04T09:46:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.