Permit: Permission-Aware Representation Intervention for Controlled Generation in Large Language Models

• URL: http://arxiv.org/abs/2605.09480v1
• Date: Sun, 10 May 2026 11:20:48 GMT
• Title: Permit: Permission-Aware Representation Intervention for Controlled Generation in Large Language Models
• Abstract summary: Large language models (LLMs) are increasingly deployed in enterprise settings where they handle sensitive documents and user context.<n>Conventional access control regulates whether information is accessible to the model, yet leaves how the model uses that information at generation time largely unconstrained.<n>We present Permit, a novel permission-aware representation intervention framework that closes this gap by enforcing fine-grained control directly on the model's hidden states.
• Score: 11.113178145712885
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly deployed in enterprise settings where they handle sensitive documents and user context, raising acute concerns over security and controllability. Conventional access control regulates whether information is accessible to the model, yet leaves how the model uses that information at generation time largely unconstrained: once sensitive content enters the context, outputs may still drift beyond a user's authorized scope. We present Permit, a novel permission-aware representation intervention framework that closes this gap by enforcing fine-grained control directly on the model's hidden states. Through exploratory analysis, we find that permission conditions induce hidden-state shifts that are (i) separable across permissions and (ii) concentrated in a small set of dominant directions. Permit exploits this geometry in two stages: it first identifies a permission-sensitive subspace from activation differences across permission conditions, and then performs lightweight interventions within this subspace to steer generation, with two concrete instantiations (offset-based and gated). Both operate atop a frozen backbone with only a handful of permission-specific parameters, achieving precise control with minimal overhead. Experimental results demonstrate that Permit performs better than the state-of-the-art method across multiple permission settings while driving information leakage to near zero, achieving over 18% F1-score improvement with >98% fewer trainable parameters.

Related papers

• One Gate Is Not Enough: Composing Stateful Pre-Action Controls for Agentic AI [0.0]
  We show that a remediation applied by one control can change the action, evidence, or context another control evaluates, invalidating that control's earlier judgment.<n>We formalize this coupling and give a remediate-and-regate protocol that restores per-action soundness in the current bounded, idempotent setting.<n>This is a mechanism demonstration on open payload data with a synthetic metadata layer, not a claim about production prevalence.
  arXiv  Detail & Related papers  (2026-08-18T22:24:44Z)
• FAVA: Formal Authorization for Verified Agents with Evidence-Backed Permission Graphs [11.447078130049512]
  FAVA is a permission-carrying authorization framework for agent execution.<n>We evaluate FAVA across OpenAgentSafety, OctoBench, and ActPlane scenarios.
  arXiv  Detail & Related papers  (2026-07-29T09:41:07Z)
• Closed-Loop Control with Rule-Aligned Small Language Models and Multi-Agent Self-Correction [17.527470001317685]
  This work investigates whether a compact Small Language Model (SLM) can be retrained for control reasoning and embedded in a validator-guided correction loop.<n>In randomized thermal-control simulations (30 experiments with 500 steps each), the framework achieves 91.5% average action-alignment accuracy.<n>Under symbolic re-mapping, it maintains a 95% in-range rate, indicating robust physical regulation despite reduced token-level agreement.
  arXiv  Detail & Related papers  (2026-06-24T13:49:01Z)
• CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs [100.38986535324284]
  We introduce textbfCIAware-Bench, a benchmark for measuring textbfcontrol textbfintervention (CI) awareness across frontier models.<n>CIAware-Bench tests whether models can distinguish their own trajectories from those modified by a control intervention.
  arXiv  Detail & Related papers  (2026-06-09T16:24:16Z)
• What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs [52.50210189669399]
  Driving vision-language models (VLMs) must accurately understand scenes across diverse conditions defined by Operational Design Domains (ODDs)<n>We propose SliceScorer, a deterministic scoring rule for missing-slice recommendation.<n>We show that SliceNav surfaces high-risk coverage gaps more effectively than prior slice-discovery methods.
  arXiv  Detail & Related papers  (2026-06-01T03:18:01Z)
• Do Coding Agents Understand Least-Privilege Authorization? [14.240332406666779]
  We study whether current models can infer authorization boundaries themselves.<n>We show that frontier models often omit permissions required by the execution chain while also granting unused or sensitive accesses.<n>We propose Sufficiency-Tightness Decomposition, which first generates a coverage-oriented policy and then audits each granted entry for grounding and sensitivity.
  arXiv  Detail & Related papers  (2026-05-14T14:05:58Z)
• Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors [48.881343993730844]
  Safety-aligned large language models (LLMs) are increasingly deployed in real-world pipelines.<n> adversaries can distribute backdoored checkpoints that behave normally under standard evaluation but jailbreak when a hidden trigger is present.<n>Recent post-hoc weight-editing methods offer an efficient approach to injecting such backdoors.
  arXiv  Detail & Related papers  (2026-04-14T06:48:33Z)
• WASD: Locating Critical Neurons as Sufficient Conditions for Explaining and Controlling LLM Behavior [9.509326860665476]
  We propose WASD (unWeaving Actionable Sufficient Directives), a novel framework that explains model behavior by identifying sufficient neural conditions for token generation.<n>Our method represents candidate conditions as neuron-activation predicates and iteratively searches for a minimal set that guarantees the current output.<n> Experiments on SST-2 and CounterFact with the Gemma-2-2B model demonstrate that our approach produces explanations that are more stable, accurate, and concise than conventional attribution graphs.
  arXiv  Detail & Related papers  (2026-03-19T04:17:19Z)
• Steering and Rectifying Latent Representation Manifolds in Frozen Multi-modal LLMs for Video Anomaly Detection [52.5174167737992]
  Video anomaly detection (VAD) aims to identify abnormal events in videos.<n>We propose SteerVAD, which advances MLLM-based VAD by shifting from passively reading to actively steering and rectifying internal representations.<n>Our method achieves state-of-the-art performance among tuning-free approaches requiring only 1% of training data.
  arXiv  Detail & Related papers  (2026-02-27T13:48:50Z)
• Efficient Representations are Controllable Representations [0.0]
  Controlling how LLMs internally represent concepts requires sophisticated methods to first identify, then intervene on the model's existing feature geometry.<n>We finetune an LLM with a simple auxiliary loss, training 16 of its 3072 residual stream dimensions to be inert interpretability flags that simply indicate what concepts are required for generation.<n>As a result, these inert flags become genuine internal features: interpretable control switches that allow us to steer generation at inference time.
  arXiv  Detail & Related papers  (2026-02-08T05:32:02Z)
• Activation-Space Anchored Access Control for Multi-Class Permission Reasoning in Large Language Models [20.49818370567961]
  Large language models (LLMs) are increasingly deployed over knowledge bases for efficient knowledge retrieval and question answering.<n>LLMs can inadvertently answer beyond a user's permission scope, leaking sensitive content.<n>We propose Activation-space Anchored Access Control (AAAC), a training-free framework for multi-class permission control.
  arXiv  Detail & Related papers  (2026-01-20T05:57:44Z)
• Role-Conditioned Refusals: Evaluating Access Control Reasoning in Large Language Models [9.010745644432221]
  We study role-conditioned refusals, focusing on the LLM's ability to adhere to access control policies by answering when authorized and refusing when not.<n>To evaluate this behavior, we created a novel dataset that extends the Spider and BIRD text-to-shot datasets.
  arXiv  Detail & Related papers  (2025-10-09T00:28:59Z)
• GLAD: Generalizable Tuning for Vision-Language Models [41.071911050087586]
  We propose a simpler and more general framework called GLAD (Generalizable LoRA tuning with RegulArized GraDient)<n>We show that merely applying LoRA achieves performance in downstream tasks comparable to current state-of-the-art prompt-based methods.
  arXiv  Detail & Related papers  (2025-07-17T12:58:15Z)
• Joint Localization and Activation Editing for Low-Resource Fine-Tuning [73.64004083269424]
  We propose a joint localization and activation editing (JoLA) method.<n>JoLA learns (1) which heads in the Transformer to edit (2) whether the intervention should be additive, multiplicative, or both and (3) the intervention parameters themselves.<n>We demonstrate that JoLA consistently outperforms existing methods.
  arXiv  Detail & Related papers  (2025-02-03T09:13:09Z)
• Get my drift? Catching LLM Task Drift with Activation Deltas [55.75645403965326]
  Task drift allows attackers to exfiltrate data or influence the LLM's output for other users.<n>We show that a simple linear classifier can detect drift with near-perfect ROC AUC on an out-of-distribution test set.<n>We observe that this approach generalizes surprisingly well to unseen task domains, such as prompt injections, jailbreaks, and malicious instructions.
  arXiv  Detail & Related papers  (2024-06-02T16:53:21Z)
• Fine-grained Controllable Video Generation via Object Appearance and Context [74.23066823064575]
  We propose fine-grained controllable video generation (FACTOR) to achieve detailed control. FACTOR aims to control objects' appearances and context, including their location and category. Our method achieves controllability of object appearances without finetuning, which reduces the per-subject optimization efforts for the users.
  arXiv  Detail & Related papers  (2023-12-05T17:47:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.