On the Rejection Criterion for Proxy-based Test-time Alignment

• URL: http://arxiv.org/abs/2604.16146v2
• Date: Mon, 20 Apr 2026 07:53:41 GMT
• Title: On the Rejection Criterion for Proxy-based Test-time Alignment
• Abstract summary: Recent works proposed test-time alignment methods that rely on a small aligned model as a proxy.<n>We argue that the confidence criterion is ill-motivated due to linguistic phenomena like ambiguous phrasing.<n>We propose a novel rejection criterion based on a conservative confidence bet.
• Score: 6.949966663998242
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent works proposed test-time alignment methods that rely on a small aligned model as a proxy that guides the generation of a larger base (unaligned) model. The implicit reward approach skews the large model distribution, whereas the nudging approach defers the generation of the next token to the small aligned model when the large base one is unconfident about its outcome. In this work, we first show that both approaches can be reduced to sampling from similar graphical models, where they differ only in the definition of a rejection criterion (or distribution). Moreover, we argue that the confidence criterion is ill-motivated due to linguistic phenomena like ambiguous phrasing. We propose a novel rejection criterion based on a conservative confidence bet. Experimentally, our novel approach outperforms previous work on several datasets.

Related papers

• Exact Unlearning from Proxies Induces Closeness Guarantees on Approximate Unlearning [21.011990634407834]
  This paper proposes a paradigm shift linking machine unlearning directly to the structure of the data distributions.<n>We show that inferring these distributions with precision enables distilling the exact unlearning signal induced by the modeling.
  arXiv  Detail & Related papers  (2026-05-11T14:57:31Z)
• VAE-Inf: A statistically interpretable generative paradigm for imbalanced classification [8.677199689027772]
  We propose a two-stage framework to bridge the gap between generative modeling and discriminative classification.<n>For inference, we introduce a projection-based score that admits a natural hypothesis testing interpretation.<n>Experiments on diverse real-world benchmarks demonstrate that our framework achieves competitive performance against other approaches.
  arXiv  Detail & Related papers  (2026-04-28T07:50:56Z)
• Sharp Convergence Rates for Masked Diffusion Models [53.117058231393834]
  We develop a total-variation based analysis for the Euler method that overcomes limitations.<n>Our results relax assumptions on score estimation, improve parameter dependencies, and establish convergence guarantees.<n>Overall, our analysis introduces a direct TV-based error decomposition along the CTMC trajectory and a decoupling-based path-wise analysis for FHS.
  arXiv  Detail & Related papers  (2026-02-26T00:47:51Z)
• Towards Anytime-Valid Statistical Watermarking [63.02116925616554]
  We develop the first e-value-based watermarking framework, Anchored E-Watermarking, that unifies optimal sampling with anytime-valid inference.<n>Our framework can significantly enhance sample efficiency, reducing the average token budget required for detection by 13-15% relative to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-02-19T18:32:26Z)
• On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inference [71.09125259964684]
  Test-time compute (TTC) has become an increasingly prominent paradigm for enhancing large language models (LLMs)<n>We study reward-filtered sequential inference, a simple procedure that selectively incorporates only high-reward generations into the context.<n>On the theoretical side, we show that reward-filtered sequential inference yields strictly stronger guarantees than standard TTC paradigms.
  arXiv  Detail & Related papers  (2025-12-04T08:21:33Z)
• Discrete Diffusion Models: Novel Analysis and New Sampler Guarantees [70.88473359544084]
  We introduce a new analytical approach for discrete diffusion models that removes the need for regularity assumptions.<n>For the standard $tau$-leaping method, we establish convergence guarantees in KL divergence that scale linearly with vocabulary size.<n>Our approach is also more broadly applicable: it provides the first convergence guarantees for other widely used samplers.
  arXiv  Detail & Related papers  (2025-09-20T17:42:29Z)
• A Survey of Low-shot Vision-Language Model Adaptation via Representer Theorem [38.84662767814454]
  Key challenge to address under the condition of limited training data is how to fine-tune pre-trained vision-language models in a parameter-efficient manner. This paper proposes a unified computational framework to integrate existing methods together, identify their nature and support in-depth comparison. As a demonstration, we extend existing methods by modeling inter-class correlation between representers in reproducing kernel Hilbert space (RKHS)
  arXiv  Detail & Related papers  (2024-10-15T15:22:30Z)
• Source-Free Unsupervised Domain Adaptation with Hypothesis Consolidation of Prediction Rationale [53.152460508207184]
  Source-Free Unsupervised Domain Adaptation (SFUDA) is a challenging task where a model needs to be adapted to a new domain without access to target domain labels or source domain data. This paper proposes a novel approach that considers multiple prediction hypotheses for each sample and investigates the rationale behind each hypothesis. To achieve the optimal performance, we propose a three-step adaptation process: model pre-adaptation, hypothesis consolidation, and semi-supervised learning.
  arXiv  Detail & Related papers  (2024-02-02T05:53:22Z)
• Single-Model Attribution of Generative Models Through Final-Layer Inversion [16.506531590300806]
  We propose a new approach for single-model attribution in the open-world setting based on final-layer inversion and anomaly detection. We show that the utilized final-layer inversion can be reduced to a convex lasso optimization problem, making our approach theoretically sound and computationally efficient.
  arXiv  Detail & Related papers  (2023-05-26T13:06:38Z)
• Post-Selection Confidence Bounds for Prediction Performance [2.28438857884398]
  In machine learning, the selection of a promising model from a potentially large number of competing models and the assessment of its generalization performance are critical tasks. We propose an algorithm how to compute valid lower confidence bounds for multiple models that have been selected based on their prediction performances in the evaluation set.
  arXiv  Detail & Related papers  (2022-10-24T13:28:43Z)
• Posterior Coreset Construction with Kernelized Stein Discrepancy for Model-Based Reinforcement Learning [78.30395044401321]
  We develop a novel model-based approach to reinforcement learning (MBRL) It relaxes the assumptions on the target transition model to belong to a generic family of mixture models. It can achieve up-to 50 percent reduction in wall clock time in some continuous control environments.
  arXiv  Detail & Related papers  (2022-06-02T17:27:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.