Sketch-based Access Control: A Multimodal Interface for Translating User Preferences into Intent-Aligned Policies

• URL: http://arxiv.org/abs/2605.10012v1
• Date: Mon, 11 May 2026 05:39:40 GMT
• Title: Sketch-based Access Control: A Multimodal Interface for Translating User Preferences into Intent-Aligned Policies
• Abstract summary: We present Sketch-based Access Control (SBAC), a sketch-based, AI-assisted access control authoring system.<n>SBAC combines the expressive power of sketching with the interpretive capabilities of multimodal large language models (MLLMs) to support the interpretation and validation of policy specifications.
• Score: 9.431045721676204
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Developing simple and expressive access controls -- interfaces to specify policies that define who should have access to resources and under what circumstances -- is a longstanding challenge in usable security. We present Sketch-based Access Control (SBAC), a sketch-based, AI-assisted access control authoring system that combines the expressive power of sketching with the interpretive capabilities of multimodal large language models (MLLMs) to support the interpretation and validation of policy specifications as they are iteratively refined. Through a formative study with 14 participants, we identified three design requirements and developed a human-AI collaborative workflow composed of three stages -- Specify, Analyze, and Test -- enabled by the system's ability to maintain and interpret evolving access control specifications. In a user evaluation with 14 participants grounded in their real-world access control scenarios, we found the system and the workflow helped participants progressively refine initially underspecified preferences into more complete and precise policies -- surfacing gaps they had not anticipated, resolving ambiguities through dialogue, and validating policy behavior through concrete scenarios.

Related papers

• Policy Description Language for Authorization using Logic-Based Programming [0.41998444721319217]
  In the multi-layer defense, it is important to authorize accesses in fine-grained granularity to compose each layer effectively.<n>We propose a policy description language which can designate many kinds of conditions for access control.
  arXiv  Detail & Related papers  (2026-06-06T11:48:00Z)
• PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models [56.37206340175191]
  Large Language Models (LLMs) are increasingly integrated into real-world decision-making.<n>Yet, their ability to comprehend and reason about policy-related content remains underexplored.<n>We present textbftextitPolicyBench, the first large-scale cross-system benchmark (US-China) evaluating policy comprehension.
  arXiv  Detail & Related papers  (2026-04-14T17:27:50Z)
• Generation of Human Comprehensible Access Control Policies from Audit Logs [2.839132202833286]
  We develop a framework that generates human understandable natural language access control policies from logs.<n>Named LANTERN (LLM-based ABAC Natural Translation and Explanation for Rule Navigation), we have instantiated the framework as a publicly accessible web based application for our results.
  arXiv  Detail & Related papers  (2026-03-15T12:21:27Z)
• Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Users [8.54162000547885]
  We show how users can customize visual question answering (VQA) tools with a real-world conversational VQA system.<n>The system lacked verbosity controls, was limited in estimating distance in space and time, relied on inaccessible image framing, and offered little to no camera guidance.<n>We discuss how customization techniques such as prompt engineering can help participants work around these limitations.
  arXiv  Detail & Related papers  (2026-02-18T22:44:43Z)
• Taxonomy of User Needs and Actions [51.86289485979439]
  The growing ubiquity of conversational AI highlights the need for frameworks that capture users' instrumental goals and the situated, adaptive, and social practices through which they achieve them.<n>To address this gap, we introduce the Taxonomy of User Needs and Actions (TUNA), an empirically grounded framework developed through iterative qualitative analysis of 1193 human-AI conversations.<n> TUNA organizes user actions into a three-level hierarchy encompassing behaviors associated with information seeking, synthesis, procedural guidance, content creation, social interaction, and meta-conversation.
  arXiv  Detail & Related papers  (2025-10-07T17:04:42Z)
• Say What You Mean: Natural Language Access Control with Large Language Models for Internet of Things [29.816322400339228]
  We present LACE, a framework that bridges the gap between human intent and machine-enforceable logic.<n>It combines prompt-guided policy generation, retrieval-augmented reasoning, and formal validation to support expressive, interpretable, and verifiable access control.<n>LACE achieves 100% correctness in verified policy generation and up to 88% decision accuracy with 0.79 F1-score.
  arXiv  Detail & Related papers  (2025-05-28T10:59:00Z)
• A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions [51.96890647837277]
  Large Language Models (LLMs) have propelled conversational AI from traditional dialogue systems into sophisticated agents capable of autonomous actions, contextual awareness, and multi-turn interactions with users.<n>This survey paper presents a desideratum for next-generation Conversational Agents - what has been achieved, what challenges persist, and what must be done for more scalable systems that approach human-level intelligence.
  arXiv  Detail & Related papers  (2025-04-07T21:01:25Z)
• A Survey on (M)LLM-Based GUI Agents [62.57899977018417]
  Graphical User Interface (GUI) Agents have emerged as a transformative paradigm in human-computer interaction.<n>Recent advances in large language models and multimodal learning have revolutionized GUI automation across desktop, mobile, and web platforms.<n>This survey identifies key technical challenges, including accurate element localization, effective knowledge retrieval, long-horizon planning, and safety-aware execution control.
  arXiv  Detail & Related papers  (2025-03-27T17:58:31Z)
• Synthesizing Access Control Policies using Large Language Models [0.5762345156477738]
  Cloud compute systems allow administrators to write access control policies that govern access to private data.<n>While policies are written in convenient languages, such as AWS Identity and Access Management Policy Language, manually written policies often become complex and error prone.<n>In this paper, we investigate whether and how well Large Language Models (LLMs) can be used to synthesize access control policies.
  arXiv  Detail & Related papers  (2025-03-14T16:40:25Z)
• The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Data [50.544186914115045]
  TEDUO is a novel training pipeline for offline language-conditioned policy learning in symbolic environments.<n>Our approach harnesses large language models (LLMs) in a dual capacity: first, as automatization tools augmenting offline datasets with richer annotations, and second, as generalizable instruction-following agents.
  arXiv  Detail & Related papers  (2024-12-09T18:43:56Z)
• Flex: End-to-End Text-Instructed Visual Navigation from Foundation Model Features [59.892436892964376]
  We investigate the minimal data requirements and architectural adaptations necessary to achieve robust closed-loop performance with vision-based control policies.<n>Our findings are synthesized in Flex (Fly lexically), a framework that uses pre-trained Vision Language Models (VLMs) as frozen patch-wise feature extractors.<n>We demonstrate the effectiveness of this approach on a quadrotor fly-to-target task, where agents trained via behavior cloning successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2024-10-16T19:59:31Z)
• Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitation [49.43094200366251]
  We propose a novel approach for few-shot adaptation to unseen tasks that exploits the semantic understanding of task decomposition. Our method, Policy Adaptation via Language Optimization (PALO), combines a handful of demonstrations of a task with proposed language decompositions. We find that PALO is able of consistently complete long-horizon, multi-tier tasks in the real world, outperforming state of the art pre-trained generalist policies.
  arXiv  Detail & Related papers  (2024-08-29T03:03:35Z)
• Constraining Participation: Affordances of Feedback Features in Interfaces to Large Language Models [49.266685603250416]
  Large language models (LLMs) are now accessible to anyone with a computer, a web browser, and an internet connection via browser-based interfaces.<n>This article examines how interactive feedback features in ChatGPT's interface afford user participation in LLMs.
  arXiv  Detail & Related papers  (2024-08-27T13:50:37Z)
• ChatSOP: An SOP-Guided MCTS Planning Framework for Controllable LLM Dialogue Agents [52.7201882529976]
  We propose SOP-guided Monte Carlo Tree Search (MCTS) planning framework to enhance controllability of dialogue agents.<n>To enable this, we curate a dataset comprising SOP-annotated multi-scenario dialogues, generated using a semi-automated role-playing system with GPT-4o.<n>We also propose a novel method that integrates Chain of Thought reasoning with supervised fine-tuning for SOP prediction.
  arXiv  Detail & Related papers  (2024-07-04T12:23:02Z)
• An Automatic Attribute Based Access Control Policy Extraction from Access Logs [5.142415132534397]
  An attribute-based access control (ABAC) model provides a more flexible approach for addressing the authorization needs of complex and dynamic systems. We present a methodology for automatically learning ABAC policy rules from access logs of a system to simplify the policy development process.
  arXiv  Detail & Related papers  (2020-03-16T15:08:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.