Derailing Non-Answers via Logit Suppression at Output Subspace Boundaries in RLHF-Aligned Language Models

• URL: http://arxiv.org/abs/2505.23848v1
• Date: Wed, 28 May 2025 20:25:24 GMT
• Title: Derailing Non-Answers via Logit Suppression at Output Subspace Boundaries in RLHF-Aligned Language Models
• Abstract summary: We introduce a method to reduce refusal rates of large language models (LLMs) on sensitive content without modifying model weights or prompts.<n>Motivated by the observation that refusals in certain models were often preceded by the specific token sequence of a token.<n>Our findings suggest that refusal behaviors can be circumvented by blocking refusal subspaces at specific points in the generation process.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce a method to reduce refusal rates of large language models (LLMs) on sensitive content without modifying model weights or prompts. Motivated by the observation that refusals in certain models were often preceded by the specific token sequence of a token marking the beginning of the chain-of-thought (CoT) block (<think>) followed by a double newline token (\n\n), we investigate the impact of two simple formatting adjustments during generation: suppressing \n\n after <think> and suppressing the end-of-sequence token after the end of the CoT block (</think>). Our method requires no datasets, parameter changes, or training, relying solely on modifying token probabilities during generation. In our experiments with official DeepSeek-R1 distillations, these interventions increased the proportion of substantive answers to sensitive prompts without affecting performance on standard benchmarks. Our findings suggest that refusal behaviors can be circumvented by blocking refusal subspaces at specific points in the generation process.

Related papers

• FOCUS & RePAIR: Mitigating Text Degeneration via Token-Level Guidance for Pruned Large Language Models [3.1974216220560816]
  Pruning is a practical approach to compress large language models (LLMs)<n>We present a token-level analysis of this failure mode by viewing decoding as a dynamical process that enters and persists in a small set of recurrent contexts.<n>Motivated by these findings, we propose two token-level guidance objectives for post-pruning fine-tuning.
  arXiv  Detail & Related papers  (2026-08-27T06:31:18Z)
• Structuring Semantic Embeddings for Principle Evaluation: A Prototype-Guided Contrastive Learning Approach [58.15427952222424]
  General-purpose text embeddings are widely deployed for such tasks, but broad semantic similarity can place semantically similar yet task-distinct examples.<n>We introduce Prototype-Guided Contrastive Learning (PGCL), a prototype-guided geometric regularization module built on top of frozen text embeddings.
  arXiv  Detail & Related papers  (2026-08-15T13:19:25Z)
• Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics [49.443264461057645]
  Diffusion and continuous flow-based language models have emerged as the leading non-autoregressive alternatives to language modeling.<n>Progress in both paradigms is overwhelmingly tracked by generative perplexity (gen-PPL): the per-token negative log-likelihood of samples under a frozen autoregressive (AR) scorer.<n>We argue that this metric is unsound. By construction, gen-PPL measures only predictability under the scoring AR, not grammaticality or semantic coherence.
  arXiv  Detail & Related papers  (2026-06-07T02:35:56Z)
• SPOT: Span-level Pause-of-Thought for Efficient and Interpretable Latent Reasoning in Large Language Models [15.95627037350657]
  Explicit Chain-of-Thought (CoT) computation incurs high inference cost due to internalize token-level traces.<n>We propose SPOT, a flexible framework that compresses explicit CoT into compact latent pause tokens without enforcing a fixed response template.<n>Experiments on reasoning benchmarks demonstrate that SPOT improves accuracy by 2.3 points on average while reducing generated tokens by 37.5%.
  arXiv  Detail & Related papers  (2026-03-06T12:34:27Z)
• CORE: Context-Robust Remasking for Diffusion Language Models [51.59514489363897]
  We propose Context-Robust Remasking (CORE), a training-free framework for inference-time revision.<n>Rather than trusting static token probabilities, CORE identifies context-brittle tokens by probing their sensitivity to targeted masked-context perturbations.<n>On LLaDA-8B-Base, CORE delivers consistent improvements across reasoning and code benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.
  arXiv  Detail & Related papers  (2026-02-04T00:12:30Z)
• PROMISE: Process Reward Models Unlock Test-Time Scaling Laws in Generative Recommendations [52.67948063133533]
  Generative Recommendation has emerged as a promising paradigm, reformulating recommendation as a sequence-to-sequence generation task over hierarchical Semantic IDs.<n>Existing methods suffer from a critical issue we term Semantic Drift, where errors in early, high-level tokens irreversibly divert the generation trajectory into irrelevant semantic subspaces.<n>We propose Promise, a novel framework that integrates dense, step-by-step verification into generative models.
  arXiv  Detail & Related papers  (2026-01-08T07:38:46Z)
• Beyond Token Length: Step Pruner for Efficient and Accurate Reasoning in Large Language Models [26.88030285500965]
  Large Reasoning Models (LRMs) demonstrate strong performance on complex tasks but often suffer from excessive verbosity, known as "overthinking"<n>We introduce textbfStep Pruner (SP), an RL framework that steers LRMs toward more efficient reasoning by favoring compact reasoning steps.<n>Our step-aware reward function prioritizes correctness while imposing penalties for redundant steps, and withholds rewards for incorrect responses to prevent the reinforcement of erroneous reasoning.
  arXiv  Detail & Related papers  (2025-10-04T13:24:26Z)
• AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models [62.70575022567081]
  We propose AdvChain, an alignment paradigm that teaches models dynamic self-correction through adversarial CoT tuning.<n>Our work establishes a new direction for building more robust and reliable reasoning models.
  arXiv  Detail & Related papers  (2025-09-29T04:27:23Z)
• Adversarial Manipulation of Reasoning Models using Internal Representations [1.024113475677323]
  We identify a linear direction in activation space during CoT token generation that predicts whether the model will refuse or comply.<n>We show that intervening only on CoT token activations suffices to control final outputs, and that incorporating this direction into prompt-based attacks improves success rates.<n>Our findings suggest that the chain-of-thought itself is a promising new target for adversarial manipulation in reasoning models.
  arXiv  Detail & Related papers  (2025-07-03T20:51:32Z)
• ConciseHint: Boosting Efficient Reasoning via Continuous Concise Hints during Generation [53.149817480019834]
  Recent advancements in large reasoning models (LRMs) have achieved notable performance enhancements on complex reasoning tasks by scaling up the generation length by Chain-of-Thought (CoT)<n>We propose a framework dubbed ConciseHint, which continuously encourages the reasoning model to speak concisely by injecting the textual hint during the token generation of the reasoning process.<n>Experiments on the state-of-the-art LRMs, including DeepSeek-R1 and Qwen-3 series, demonstrate that our method can effectively produce concise reasoning processes while maintaining performance well.
  arXiv  Detail & Related papers  (2025-06-23T16:20:44Z)
• Normalized Attention Guidance: Universal Negative Guidance for Diffusion Model [57.20761595019967]
  We present Normalized Attention Guidance (NAG), an efficient, training-free mechanism that applies extrapolation in attention space with L1-based normalization and refinement.<n>NAG restores effective negative guidance where CFG collapses while maintaining fidelity.<n>NAG generalizes across architectures (UNet, DiT), sampling regimes (few-step, multi-step), and modalities (image, video)
  arXiv  Detail & Related papers  (2025-05-27T13:30:46Z)
• Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Sampling [90.86991492288487]
  evaluating constraint on every token can be prohibitively expensive.<n> LCD can distort the global distribution over strings, sampling tokens based only on local information.<n>We show that our approach is superior to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-04-07T18:30:18Z)
• (G)I-DLE: Generative Inference via Distribution-preserving Logit Exclusion with KL Divergence Minimization for Constrained Decoding [0.0]
  (G)I-DLE is a new approach to constrained decoding that leverages KL divergence to preserve the intrinsic conditional probability distribution of autoregressive language models.<n>We validate our method on the K2-Eval dataset, specifically designed to assess Korean language fluency.
  arXiv  Detail & Related papers  (2025-03-23T12:37:14Z)
• Non-Halting Queries: Exploiting Fixed Points in LLMs [4.091772241106195]
  We introduce a new vulnerability that exploits fixed points in autoregressive models and use it to craft queries that never halt.<n>We rigorously analyze the conditions under which the non-halting anomaly presents itself.<n>We demonstrate non-halting queries in many experiments performed in base unaligned models.
  arXiv  Detail & Related papers  (2024-10-08T18:38:32Z)
• AdjointDPM: Adjoint Sensitivity Method for Gradient Backpropagation of Diffusion Probabilistic Models [103.41269503488546]
  Existing customization methods require access to multiple reference examples to align pre-trained diffusion probabilistic models with user-provided concepts. This paper aims to address the challenge of DPM customization when the only available supervision is a differentiable metric defined on the generated contents. We propose a novel method AdjointDPM, which first generates new samples from diffusion models by solving the corresponding probability-flow ODEs. It then uses the adjoint sensitivity method to backpropagate the gradients of the loss to the models' parameters.
  arXiv  Detail & Related papers  (2023-07-20T09:06:21Z)
• Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation [92.42032403795879]
  We show that pretrained language models (LMs) such as GPT2 still tend to generate repetitive texts. We attribute their overestimation of token-level repetition probabilities to the learning bias. We find that LMs use longer-range dependencies to predict repetitive tokens than non-repetitive ones, which may be the cause of sentence-level repetition loops.
  arXiv  Detail & Related papers  (2023-07-04T07:53:55Z)
• Elastic Weight Removal for Faithful and Abstractive Dialogue Generation [61.40951756070646]
  A dialogue system should generate responses that are faithful to the knowledge contained in relevant documents. Many models generate hallucinated responses instead that contradict it or contain unverifiable information. We show that our method can be extended to simultaneously discourage hallucinations and extractive responses.
  arXiv  Detail & Related papers  (2023-03-30T17:40:30Z)
• Explicit Regularization in Overparametrized Models via Noise Injection [14.492434617004932]
  We show that small perturbations induce explicit regularization for simple finite-dimensional models. We empirically show that the small perturbations lead to better generalization performance than vanilla (stochastic) gradient descent training.
  arXiv  Detail & Related papers  (2022-06-09T17:00:23Z)
• BSN++: Complementary Boundary Regressor with Scale-Balanced Relation Modeling for Temporal Action Proposal Generation [85.13713217986738]
  We present BSN++, a new framework which exploits complementary boundary regressor and relation modeling for temporal proposal generation. Not surprisingly, the proposed BSN++ ranked 1st place in the CVPR19 - ActivityNet challenge leaderboard on temporal action localization task.
  arXiv  Detail & Related papers  (2020-09-15T07:08:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.