WARP: Guaranteed Inner-Layer Repair of NLP Transformers

• URL: http://arxiv.org/abs/2604.00938v1
• Date: Wed, 01 Apr 2026 14:12:49 GMT
• Title: WARP: Guaranteed Inner-Layer Repair of NLP Transformers
• Abstract summary: We present WARP, a constraint-based repair framework that extends repair beyond the last layer of Transformer models.<n> WARP formulates repair as a convex quadratic program derived from a first-order linearization of the logit gap.<n>We show that WARP converges to solutions satisfying all repair constraints under mild assumptions.
• Score: 4.191577542171072
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer-based NLP models remain vulnerable to adversarial perturbations, yet existing repair methods face a fundamental trade-off: gradient-based approaches offer flexibility but lack verifiability and often overfit; methods that do provide repair guarantees are restricted to the final layer or small networks, significantly limiting the parameter search space available for repair. We present WARP (Weight-Adjusted Repair with Provability), a constraint-based repair framework that extends repair beyond the last layer of Transformer models. WARP formulates repair as a convex quadratic program derived from a first-order linearization of the logit gap, enabling tractable optimization over a high-dimensional parameter space. Under the condition that the first-order approximation holds, this formulation induces three per-sample guarantees: (i) a positive margin constraint ensuring correct classification on repaired inputs, (ii) preservation constraints over a designated remain set, and (iii) a certified robustness radius derived from Lipschitz continuity. To ensure feasibility across varying model architectures, we introduce a sensitivity-based preprocessing step that conditions the optimization landscape accordingly. We further show that the iterative optimization procedure converges to solutions satisfying all repair constraints under mild assumptions. Empirical evaluation on encoder-only Transformers with varying layer architectures validates that these guarantees hold in practice while improving robustness to adversarial inputs. Our results demonstrate that guaranteed, generalizable Transformer repair is achievable through principled constraint-based optimization.

Related papers

• Reproducing FACTER: Fairness via Conformal Thresholding and Prompt Repair [0.6977547772245161]
  FACTER is a model-agnostic framework designed to jointly enforce fairness and statistical coverage in recommendation.<n>We conduct a study of the FACTER framework across diverse architectures and dataset sparsity levels.<n>Our analysis shows that FACTER consistently reduces adaptive-threshold violation counts, but that these reductions are not consistently reflected in global fairness metrics.
  arXiv  Detail & Related papers  (2026-06-26T21:37:01Z)
• Improving Visual Representation Alignment Generation with GRPO [51.071351994330605]
  diffusion transformers have demonstrated strong image synthesis capabilities but remain inefficient to train due to weak alignment.<n>We propose VRPO, a reinforcement-based optimization strategy that replaces REPA's static alignment loss with a generative representation policy optimization objective.<n>Our VRPO-driven training seamlessly integrates into diffusion transformers, introducing negligible cost and preserving full compatibility with SiT and DiT architectures.
  arXiv  Detail & Related papers  (2026-05-30T07:21:40Z)
• AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generation [59.964989458924585]
  AdaExplore is an agent framework that enables self-improvement via accumulated execution feedback for performance-critical kernel code generation.<n>In the adaptation stage, the agent synthesizes tasks and converts recurring failures into a reusable memory of validity rules.<n>In the search stage, the agent organizes candidate kernels as a tree and alternates between small local refinements and larger structural regeneration.
  arXiv  Detail & Related papers  (2026-04-17T18:25:03Z)
• Beyond Fixed Tests: Repository-Level Issue Resolution as Coevolution of Code and Behavioral Constraints [17.818522356206977]
  Most large language model (LLM)-based repair systems treat behavioral constraints as fixed during repair.<n>We propose Agent-CoEvo, a coevolutionary multi-agent framework in which candidate code patches and test patches are jointly explored and refined.<n>Agent-CoEvo consistently outperforms state-of-the-art agent-based and agentless baselines in both repair success and test reproduction quality.
  arXiv  Detail & Related papers  (2026-04-06T10:26:46Z)
• SAHOO: Safeguarded Alignment for High-Order Optimization Objectives in Recursive Self-Improvement [16.505918019260964]
  SAHOO is a framework to monitor and control drift through three safeguards.<n>GDI is a multi-signal detector combining semantic, lexical, structural, and distributional measures.<n> constraint preservation checks enforce safety-critical invariants such as syntactic correctness and non-hallucination.
  arXiv  Detail & Related papers  (2026-03-06T14:44:51Z)
• Score-Guided Proximal Projection: A Unified Geometric Framework for Rectified Flow Editing [1.0312968200748118]
  Rectified Flow models achieve state-of-the-art generation quality, yet controlling them for precise tasks remains a challenge.<n>Current approaches bifurcate into inversion-based guidance, which suffers from "geometric locking"<n>We propose Score-Guided Proximal Projection, a unified framework that bridges the gap between deterministic optimization and contraction sampling.
  arXiv  Detail & Related papers  (2026-03-05T23:44:45Z)
• A Constrained Optimization Perspective of Unrolled Transformers [77.12297732942095]
  We introduce a constrained optimization framework for training transformers that behave like optimization descent algorithms.<n>We observe constrained transformers achieve stronger to perturbations robustness and maintain higher out-of-distribution generalization.
  arXiv  Detail & Related papers  (2026-01-24T02:12:39Z)
• Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality [53.525547349715595]
  We propose a novel primal-only algorithm called Rectified Robust Policy Optimization (RRPO)<n>RRPO operates directly on the primal problem without relying on dual formulations.<n>We show convergence to an approximately optimal feasible policy with complexity matching the best-known lower bound.
  arXiv  Detail & Related papers  (2025-08-24T16:59:38Z)
• Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations [67.35596444651037]
  Vision-language models (VLMs) exhibit remarkable zero-shot capabilities but struggle with distribution shifts in downstream tasks when labeled data is unavailable.<n>We propose a Reliable Test-time Adaptation (ReTA) method that enhances reliability from two perspectives.
  arXiv  Detail & Related papers  (2025-07-13T05:37:33Z)
• Rectifying Conformity Scores for Better Conditional Coverage [75.73184036344908]
  We present a new method for generating confidence sets within the split conformal prediction framework.<n>Our method performs a trainable transformation of any given conformity score to improve conditional coverage while ensuring exact marginal coverage.
  arXiv  Detail & Related papers  (2025-02-22T19:54:14Z)
• Double Duality: Variational Primal-Dual Policy Optimization for Constrained Reinforcement Learning [132.7040981721302]
  We study the Constrained Convex Decision Process (MDP), where the goal is to minimize a convex functional of the visitation measure. Design algorithms for a constrained convex MDP faces several challenges, including handling the large state space.
  arXiv  Detail & Related papers  (2024-02-16T16:35:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.