Amplification Effects in Test-Time Reinforcement Learning: Safety and Reasoning Vulnerabilities

• URL: http://arxiv.org/abs/2603.15417v1
• Date: Mon, 16 Mar 2026 15:28:59 GMT
• Title: Amplification Effects in Test-Time Reinforcement Learning: Safety and Reasoning Vulnerabilities
• Abstract summary: Test-time training (TTT) has emerged as a promising method to improve the reasoning abilities of large language models.<n>We show that TTT methods that enhance LLM reasoning by promoting self-consistency can lead to amplification behaviors and reasoning degradation.
• Score: 23.619172971228853
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Test-time training (TTT) has recently emerged as a promising method to improve the reasoning abilities of large language models (LLMs), in which the model directly learns from test data without access to labels. However, this reliance on test data also makes TTT methods vulnerable to harmful prompt injections. In this paper, we investigate safety vulnerabilities of TTT methods, where we study a representative self-consistency-based test-time learning method: test-time reinforcement learning (TTRL), a recent TTT method that improves LLM reasoning by rewarding self-consistency using majority vote as a reward signal. We show that harmful prompt injection during TTRL amplifies the model's existing behaviors, i.e., safety amplification when the base model is relatively safe, and harmfulness amplification when it is vulnerable to the injected data. In both cases, there is a decline in reasoning ability, which we refer to as the reasoning tax. We also show that TTT methods such as TTRL can be exploited adversarially using specially designed "HarmInject" prompts to force the model to answer jailbreak and reasoning queries together, resulting in stronger harmfulness amplification. Overall, our results highlight that TTT methods that enhance LLM reasoning by promoting self-consistency can lead to amplification behaviors and reasoning degradation, highlighting the need for safer TTT methods.

Related papers

• Test-Time Training Undermines Safety Guardrails [53.00299679873018]
  Test-Time Training (TTT) is an emerging paradigm that enables models to adapt their parameters during inference.<n>We identify three threat models for TTT and demonstrate how attackers can leverage them to bypass safety filters.
  arXiv  Detail & Related papers  (2026-05-21T19:27:06Z)
• What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time [57.533031432715084]
  Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams.<n>Existing TTRL methods rely exclusively on positive pseudo-labeling strategies.<n>We propose SCRL, a robust test-time reinforcement learning framework that effectively mitigates label noise amplification.
  arXiv  Detail & Related papers  (2026-03-20T11:47:12Z)
• From the Inside Out: Progressive Distribution Refinement for Confidence Calibration [11.940520783793337]
  Leveraging the model's internal information as the self-reward signal in Reinforcement Learning (RL) has received extensive attention due to its label-free nature.<n>We propose DistriTTRL, which leverages the distribution prior to the model's confidence during RL to progressively optimize the reward signal.<n>We mitigate the phenomenon of consistent reward hacking caused by the voting-based TTS strategies through diversity-targeted penalties.
  arXiv  Detail & Related papers  (2026-03-17T13:26:29Z)
• Boosting ASR Robustness via Test-Time Reinforcement Learning with Audio-Text Semantic Rewards [8.109014000578766]
  We present ASR-TRA, a novel Testtime Reinforcement Adaptation framework inspired by causal intervention.<n>Our method achieves higher accuracy while maintaining lower latency than existing TTA baselines.<n>Our approach provides a practical and robust solution for deploying ASR systems in challenging real-world conditions.
  arXiv  Detail & Related papers  (2026-03-05T14:43:15Z)
• Tool Verification for Test-Time Reinforcement Learning [70.09740926883818]
  Test-time reinforcement learning (TTRL) has emerged as a promising paradigm for self-evolving large reasoning models.<n>We present T3RL (Tool-Verification for Test-Time Reinforcement Learning), which introduces test-time tool verification into reward estimation.
  arXiv  Detail & Related papers  (2026-03-02T18:57:52Z)
• AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models [62.70575022567081]
  We propose AdvChain, an alignment paradigm that teaches models dynamic self-correction through adversarial CoT tuning.<n>Our work establishes a new direction for building more robust and reliable reasoning models.
  arXiv  Detail & Related papers  (2025-09-29T04:27:23Z)
• Test-Time Training Provably Improves Transformers as In-context Learners [49.09821664572445]
  We investigate a gradient-based TTT algorithm for in-context learning.<n>We train a transformer model on the in-context demonstrations provided in the test prompt.<n>As our empirical contribution, we study the benefits of TTT for TabPFN.
  arXiv  Detail & Related papers  (2025-03-14T20:06:37Z)
• Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection [36.407171992845456]
  We introduce adaptive learn-then-test (aLTT) to provide finite-sample statistical guarantees on the population risk of AI models.<n>Unlike the existing learn-then-test (LTT) technique, aLTT implements sequential data-dependent multiple hypothesis testing (MHT) with early termination by leveraging e-processes.
  arXiv  Detail & Related papers  (2024-09-24T08:14:26Z)
• Active Test-Time Adaptation: Theoretical Analyses and An Algorithm [51.84691955495693]
  Test-time adaptation (TTA) addresses distribution shifts for streaming test data in unsupervised settings. We propose the novel problem setting of active test-time adaptation (ATTA) that integrates active learning within the fully TTA setting.
  arXiv  Detail & Related papers  (2024-04-07T22:31:34Z)
• Test-Time Adaptation with Perturbation Consistency Learning [32.58879780726279]
  We propose a simple test-time adaptation method to promote the model to make stable predictions for samples with distribution shifts. Our method can achieve higher or comparable performance with less inference time over strong PLM backbones.
  arXiv  Detail & Related papers  (2023-04-25T12:29:22Z)
• Uncovering Adversarial Risks of Test-Time Adaptation [41.19226800089764]
  Test-time adaptation (TTA) has been proposed as a promising solution for addressing distribution shifts. We uncover a novel security vulnerability of TTA based on the insight that predictions on benign samples can be impacted by malicious samples in the same batch. We propose Distribution Invading Attack (DIA), which injects a small fraction of malicious data into the test batch.
  arXiv  Detail & Related papers  (2023-01-29T22:58:05Z)
• Adversarial Attacks and Defense for Non-Parametric Two-Sample Tests [73.32304304788838]
  This paper systematically uncovers the failure mode of non-parametric TSTs through adversarial attacks. To enable TST-agnostic attacks, we propose an ensemble attack framework that jointly minimizes the different types of test criteria. To robustify TSTs, we propose a max-min optimization that iteratively generates adversarial pairs to train the deep kernels.
  arXiv  Detail & Related papers  (2022-02-07T11:18:04Z)
• Enhancing Adversarial Robustness via Test-time Transformation Ensembling [51.51139269928358]
  We show how equipping models with Test-time Transformation Ensembling can work as a reliable defense against adversarial attacks. We show that TTE consistently improves model robustness against a variety of powerful attacks without any need for re-training.
  arXiv  Detail & Related papers  (2021-07-29T15:32:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.