Models Know Their Shortcuts: Deployment-Time Shortcut Mitigation

• URL: http://arxiv.org/abs/2604.12277v1
• Date: Tue, 14 Apr 2026 04:43:29 GMT
• Title: Models Know Their Shortcuts: Deployment-Time Shortcut Mitigation
• Abstract summary: Shortcut Guardrail mitigates token-level shortcuts without access to the original training data or shortcut annotations.<n>Shortcut Guardrail improves overall accuracy and worst-group accuracy over the unmitigated model under distribution shifts.
• Score: 5.949467977832433
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Pretrained language models often rely on superficial features that appear predictive during training yet fail to generalize at test time, a phenomenon known as shortcut learning. Existing mitigation methods generally operate at training time and require heavy supervision such as access to the original training data or prior knowledge of shortcut type. We propose Shortcut Guardrail, a deployment-time framework that mitigates token-level shortcuts without access to the original training data or shortcut annotations. Our key insight is that gradient-based attribution on a biased model highlights shortcut tokens. Building on this finding, we train a lightweight LoRA-based debiasing module with a Masked Contrastive Learning (MaskCL) objective that encourages consistent representations with or without individual tokens. Across sentiment classification, toxicity detection, and natural language inference under both naturally occurring and controlled shortcuts, Shortcut Guardrail improves overall accuracy and worst-group accuracy over the unmitigated model under distribution shifts while preserving in-distribution performance.

Related papers

• Suppressing Forgery-Specific Shortcuts for Generalizable Deepfake Detection [59.056076147081775]
  Deepfake detection suffers from poor generalization across forgery methods.<n>Recent approaches attempt to improve generalization, but lack explicit mechanism to identify and suppress such shortcuts.<n>We propose Shortcut Subspace Suppression (S3) framework that explicitly characterizes and suppresses method-specific shortcuts.
  arXiv  Detail & Related papers  (2026-06-01T07:54:58Z)
• Mitigating Shortcut Reasoning in Language Models: A Gradient-Aware Training Approach [31.233459232809608]
  Shortcut-Aware Reasoning Training (SART) is a gradient-aware framework that detects and mitigates shortcut-promoting samples.<n>Our method identifies shortcut signals through gradient misalignment with validation objectives and answer-token concentration.
  arXiv  Detail & Related papers  (2026-03-21T18:17:17Z)
• Single-weight Model Editing for Post-hoc Spurious Correlation Neutralization [54.8794775172033]
  Neural network training tends to exploit the simplest features as shortcuts to greedily minimize training loss.<n>Some of these features might be spuriously correlated with the target labels, leading to incorrect predictions by the model.<n>We propose a unique precise class removal technique that makes a single-weight modification, which entails negligible performance compromise.
  arXiv  Detail & Related papers  (2025-01-24T02:22:42Z)
• On the Shortcut Learning in Multilingual Neural Machine Translation [95.30470845501141]
  This study revisits the commonly-cited off-target issue in multilingual neural machine translation (MNMT) We attribute the off-target issue to the overfitting of the shortcuts of (non-centric, centric) language mappings. Analyses on learning dynamics show that the shortcut learning generally occurs in the later stage of model training.
  arXiv  Detail & Related papers  (2024-11-15T21:09:36Z)
• Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models [20.70050968223901]
  This study addresses the overlooked impact of subtler, more complex shortcuts that compromise model reliability beyond oversimplified shortcuts. We introduce a comprehensive benchmark that categorizes shortcuts into occurrence, style, and concept. Our research systematically investigates models' resilience and susceptibilities to sophisticated shortcuts.
  arXiv  Detail & Related papers  (2024-09-26T01:17:42Z)
• The Right Time Matters: Data Arrangement Affects Zero-Shot Generalization in Instruction Tuning [86.19804569376333]
  We show that zero-shot generalization happens very early during instruction tuning.<n>We propose a more grounded training data arrangement framework, Test-centric Multi-turn Arrangement.
  arXiv  Detail & Related papers  (2024-06-17T16:40:21Z)
• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• Understanding prompt engineering may not require rethinking generalization [56.38207873589642]
  We show that the discrete nature of prompts, combined with a PAC-Bayes prior given by a language model, results in generalization bounds that are remarkably tight by the standards of the literature. This work provides a possible justification for the widespread practice of prompt engineering.
  arXiv  Detail & Related papers  (2023-10-06T00:52:48Z)
• Self-regulating Prompts: Foundational Model Adaptation without Forgetting [112.66832145320434]
  We introduce a self-regularization framework for prompting called PromptSRC. PromptSRC guides the prompts to optimize for both task-specific and task-agnostic general representations.
  arXiv  Detail & Related papers  (2023-07-13T17:59:35Z)
• How to Construct Perfect and Worse-than-Coin-Flip Spoofing Countermeasures: A Word of Warning on Shortcut Learning [20.486639064376014]
  Shortcut learning, or Clever Hans effect refers to situations where a learning agent learns spurious correlations present in data, resulting in biased models. We focus on finding shortcuts in deep learning based spoofing countermeasures (CMs) that predict whether a given utterance is spoofed or not.
  arXiv  Detail & Related papers  (2023-05-31T15:58:37Z)
• Chroma-VAE: Mitigating Shortcut Learning with Generative Classifiers [44.97660597940641]
  We show that generative models alone are not sufficient to prevent shortcut learning. In particular, we propose Chroma-VAE, a two-pronged approach where a VAE is initially trained to isolate the shortcut in a small latent subspace. In addition to demonstrating the efficacy of Chroma-VAE on benchmark and real-world shortcut learning tasks, our work highlights the potential for manipulating the latent space of generative classifiers to isolate or interpret specific correlations.
  arXiv  Detail & Related papers  (2022-11-28T11:27:50Z)
• Learning Less Generalizable Patterns with an Asymmetrically Trained Double Classifier for Better Test-Time Adaptation [4.893694715581673]
  We propose a novel approach using a pair of classifiers and a shortcut patterns avoidance loss. Our method improves upon the state-of-the-art results on both benchmarks and benefits the most to test-time batch normalization.
  arXiv  Detail & Related papers  (2022-10-17T08:05:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.