Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projection

• URL: http://arxiv.org/abs/2603.14825v1
• Date: Mon, 16 Mar 2026 04:59:24 GMT
• Title: Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projection
• Abstract summary: We investigate whether safety and utility are inherently antagonistic objectives.<n>We focus on a modality induced bias direction consistently observed across datasets.<n>We propose Two Birds, One Projection, an efficient inference time jailbreak defence.
• Score: 6.473196638281021
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing jailbreak defence frameworks for Large Vision-Language Models often suffer from a safety utility tradeoff, where strengthening safety inadvertently degrades performance on general visual-grounded reasoning tasks. In this work, we investigate whether safety and utility are inherently antagonistic objectives. We focus on a modality induced bias direction consistently observed across datasets, which arises from suboptimal coupling between the Large Language Model backbone and visual encoders. We further demonstrate that this direction undermines performance on both tasks. Leveraging this insight, we propose Two Birds, One Projection, an efficient inference time jailbreak defence that projects cross-modal features onto the null space of the identified bias direction to remove the corresponding components. Requiring only a single forward pass, our method effectively breaks the conventional tradeoff, simultaneously improving both safety and utility across diverse benchmarks.

Related papers

• Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model [73.59026525350808]
  We propose a dual adversarial fine-tuning framework that jointly optimize visual and semantic supervision signals from two modalities.<n>Our approach outperforms the state-of-the-art method in adversarial robustness evaluation across zero-shot classification, image captioning, and visual question answering tasks.
  arXiv  Detail & Related papers  (2026-07-21T10:49:29Z)
• Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection [21.411964269762432]
  We present a comprehensive framework for diagnosing and repairing unsafe channels within Large Vision-Language Models (LVLMs)<n>We first perform causal mediation analysis to identify neurons and layers that are causally responsible for unsafe behaviors.<n>We introduce a dual-modal safety subspace projection method that learns generalized safety subspaces for both visual and textual modalities.
  arXiv  Detail & Related papers  (2026-03-28T11:31:16Z)
• Enhancing Safety of Large Language Models via Embedding Space Separation [24.821172423424866]
  Large language models (LLMs) have achieved impressive capabilities, yet ensuring their safety against harmful prompts remains a critical challenge.<n>Recent work has revealed that the latent representations (embeddings) of harmful and safe queries in LLMs typically exhibit linear separability.<n>Motivated by this observation, we propose a representation-level fine-tuning approach, named Embedding Space Separation (ES2)
  arXiv  Detail & Related papers  (2026-03-01T07:22:38Z)
• Self-Guard: Defending Large Reasoning Models via enhanced self-reflection [54.775612141528164]
  Self-Guard is a lightweight safety defense framework for Large Reasoning Models.<n>It bridges the awareness-compliance gap, achieving robust safety performance without compromising model utility.<n>Self-Guard exhibits strong generalization across diverse unseen risks and varying model scales.
  arXiv  Detail & Related papers  (2026-01-31T13:06:11Z)
• Attributing and Exploiting Safety Vectors through Global Optimization in Large Language Models [50.91504059485288]
  We propose a framework that identifies safety-critical attention heads through global optimization over all heads simultaneously.<n>We develop a novel inference-time white-box jailbreak method that exploits the identified safety vectors through activation repatching.
  arXiv  Detail & Related papers  (2026-01-22T09:32:43Z)
• Think-Reflect-Revise: A Policy-Guided Reflective Framework for Safety Alignment in Large Vision Language Models [58.17589701432514]
  Think-Reflect-Revise (TRR) is a training framework designed to enhance the safety alignment of Large Vision Language Models (LVLMs)<n>We first build a Reflective Safety Reasoning (ReSafe) dataset with 5,000 examples that follow a think-reflect-revise process.<n>We then fine-tune the target model using the ReSafe dataset to initialize reflective behavior, and finally reinforce policy-guided reflection through reinforcement learning.
  arXiv  Detail & Related papers  (2025-12-08T03:46:03Z)
• DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Models [50.21378052667732]
  We conduct an in-depth analysis of dLLM vulnerabilities to jailbreak attacks across two distinct dimensions: intra-step and inter-step dynamics.<n>We propose DiffuGuard, a training-free defense framework that addresses vulnerabilities through a dual-stage approach.
  arXiv  Detail & Related papers  (2025-09-29T05:17:10Z)
• LatentGuard: Controllable Latent Steering for Robust Refusal of Attacks and Reliable Response Generation [4.29885665563186]
  LATENTGUARD is a framework that combines behavioral alignment with supervised latent space control for interpretable and precise safety steering.<n>Our results show significant improvements in both safety controllability and response interpretability without compromising utility.
  arXiv  Detail & Related papers  (2025-09-24T07:31:54Z)
• SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoning [51.78514648677898]
  We propose SAFER, a framework for Safety Alignment via eFficient Ex-Ante Reasoning.<n>Our approach instantiates structured Ex-Ante reasoning through initial assessment, rule verification, and path calibration.<n> Experiments on multiple open-source LLMs demonstrate that SAFER significantly enhances safety performance while maintaining helpfulness and response efficiency.
  arXiv  Detail & Related papers  (2025-04-03T16:07:38Z)
• CeTAD: Towards Certified Toxicity-Aware Distance in Vision Language Models [16.5022773312661]
  We propose a universal certified defence framework to safeguard large vision-language models against jailbreak attacks.<n>First, we proposed a novel distance metric to quantify semantic discrepancies between malicious and intended responses.<n>Then, we devise a regressed certification approach that employs randomized smoothing to provide formal robustness guarantees.
  arXiv  Detail & Related papers  (2025-03-08T17:33:55Z)
• Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking [54.10710423370126]
  We propose Reasoning-to-Defend (R2D), a training paradigm that integrates a safety-aware reasoning mechanism into Large Language Models' generation process.<n>CPO enhances the model's perception of the safety status of given dialogues.<n>Experiments demonstrate that R2D effectively mitigates various attacks and improves overall safety, while maintaining the original performances.
  arXiv  Detail & Related papers  (2025-02-18T15:48:46Z)
• Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models [25.606641582511106]
  We propose a novel dataset that integrates multi-image inputs with safety Chain-of-Thought (CoT) labels as fine-grained reasoning logic to improve model performance.<n>Our experiments demonstrate that fine-tuning InternVL2.5-8B with MIS significantly outperforms both powerful open-source models and API-based models in challenging multi-image tasks.
  arXiv  Detail & Related papers  (2025-01-30T17:59:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.