PhySE: A Psychological Framework for Real-Time AR-LLM Social Engineering Attacks

• URL: http://arxiv.org/abs/2604.23148v1
• Date: Sat, 25 Apr 2026 05:31:40 GMT
• Title: PhySE: A Psychological Framework for Real-Time AR-LLM Social Engineering Attacks
• Abstract summary: Malicious actors use Augmented Reality (AR) glasses to capture a target visual and vocal data.<n>Large Language Model (LLM) analyzes this data to identify the individual and generate a detailed social profile.<n>LLM-powered agents employ social engineering strategies, providing real-time conversation suggestions to gain the target trust.
• Score: 11.801855344327928
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The emerging threat of AR-LLM-based Social Engineering (AR-LLM-SE) attacks (e.g. SEAR) poses a significant risk to real-world social interactions. In such an attack, a malicious actor uses Augmented Reality (AR) glasses to capture a target visual and vocal data. A Large Language Model (LLM) then analyzes this data to identify the individual and generate a detailed social profile. Subsequently, LLM-powered agents employ social engineering strategies, providing real-time conversation suggestions, to gain the target trust and ultimately execute phishing or other malicious acts. Despite its potential, the practical application of AR-LLM-SE faces two major bottlenecks, (1) Cold-start personalization, Current Retrieval-Augmented Generation (RAG) methods introduce critical delays in the earliest turns, slowing initial profile formation and disrupting real-time interaction, (2) Static Attack Strategies, Existing approaches rely on fixed-stage, handcrafted social engineering tactics that lack foundation in established psychological theory. To address these limitations, we propose PhySE, a novel framework with two core innovations, (1) VLM-Based SocialContext Training, To eliminate profiling delays, we efficiently pre-train a Visual Language Model (VLM) with social-context data, enabling rapid, on-the-fly profile generation, (2) Adaptive Psychological Agent, We introduce a psychological LLM that dynamically deploys distinct classes of psychological strategies based on target response, moving beyond static, handcrafted scripts. We evaluated PhySE through an IRB-approved user study with 60 participants, collecting a novel dataset of 360 annotated conversations across diverse social scenarios.

Related papers

• SocialRL: Refining LLMs' Social Intelligence through Multi-turn Reinforcement Learning and Reward Design [52.69818441707104]
  Social intelligence enables agents to read social context, infer intent, and adapt over sustained dialogue.<n>Existing reinforcement learning methods optimize single-turn utterances and sparse outcome rewards.<n>We propose SocialRL, a multi-turn reinforcement learning framework addressing both challenges.
  arXiv  Detail & Related papers  (2026-09-09T06:06:34Z)
• ALSO: Adversarial Online Strategy Optimization for Social Agents [63.84788516416339]
  We propose textbfALSO (textbfAdrial ontextbfLine textbfStrategy textbfOptimization) as the first framework for online strategy optimization in social simulation.<n> ALSO formulates multi-turn interaction as an adversarial bandit problem, where combinations of static personas and dynamic strategy instructions are treated as arms.<n>Also introduces a lightweight neural surrogate to predict rewards from interaction histories, enabling sample-efficient exploration and continuous online adaptation.
  arXiv  Detail & Related papers  (2026-05-15T09:25:15Z)
• UNSEEN: A Cross-Stack LLM Unlearning Defense against AR-LLM Social Engineering Attacks [18.1862287371633]
  AR-LLM-SE attack poses great threats to real-world social life.<n>In such AR-LLM-SE attack, the attacker can leverage AR (Augmented Reality) glass to capture the image and vocal information of the target.<n>We present UNSEEN, a coordinated cross-stack defense that combines an AR ACL for identity-gated sensing, F-RMU-based LLM unlearning for sensitive profile suppression, and agent guardrails for adaptive interaction control.
  arXiv  Detail & Related papers  (2026-04-25T04:49:02Z)
• Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimization [51.12422886183246]
  Large Language Models (LLMs) have developed rapidly in web services, delivering unprecedented capabilities while amplifying societal risks.<n>Existing works tend to focus on either isolated jailbreak attacks or static defenses, neglecting the dynamic interplay between evolving threats and safeguards in real-world web contexts.<n>We propose ACE-Safety, a novel framework that jointly optimize attack and defense models by seamlessly integrating two key innovative procedures.
  arXiv  Detail & Related papers  (2025-11-24T15:23:41Z)
• Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs [72.08224879435762]
  textttLearn-to-Ask is a simulator-free framework for learning and deploying proactive dialogue agents.<n>Our approach culminates in the successful deployment of LLMs into a live, large-scale online AI service.
  arXiv  Detail & Related papers  (2025-10-29T12:08:07Z)
• Searching for Privacy Risks in LLM Agents via Simulation [61.229785851581504]
  We present a search-based framework that alternates between improving attack and defense strategies through the simulation of privacy-critical agent interactions.<n>We find that attack strategies escalate from direct requests to sophisticated tactics, such as impersonation and consent forgery.<n>The discovered attacks and defenses transfer across diverse scenarios and backbone models, demonstrating strong practical utility for building privacy-aware agents.
  arXiv  Detail & Related papers  (2025-08-14T17:49:09Z)
• LLMs are Introvert [21.542534041341774]
  Large language models (LLMs) offer new potential for simulating psychological aspects of information spread.<n>Initial experiments revealed significant gaps between LLM-generated behaviors and authentic human dynamics.<n>We propose the Social Information Processing-based Chain of Thought (SIP-CoT) mechanism enhanced by emotion-guided memory.
  arXiv  Detail & Related papers  (2025-07-08T03:32:38Z)
• Automated Profile Inference with Language Model Agents [67.32226960040514]
  We study a new threat that LLMs pose to online pseudonymity, called automated profile inference.<n>An adversary can instruct LLMs to automatically scrape and extract sensitive personal attributes from publicly visible user activities on pseudonymous platforms.<n>We introduce an automated profiling framework called AutoProfiler to assess the feasibility of such threats in real-world scenarios.
  arXiv  Detail & Related papers  (2025-05-18T13:05:17Z)
• On the Feasibility of Using MultiModal LLMs to Execute AR Social Engineering Attacks [8.28564202645918]
  We propose a framework for orchestrating AR-driven Social Engineering attacks using Multimodal Large Language Models.<n>Our results show that SEAR is highly effective at eliciting high-risk behaviors.<n>We identify notable limitations such as occasionally artificial'' due to perceived authenticity gaps.
  arXiv  Detail & Related papers  (2025-04-16T05:18:36Z)
• Personalized Attacks of Social Engineering in Multi-turn Conversations: LLM Agents for Simulation and Detection [19.604708321391012]
  Social engineering (SE) attacks on social media platforms pose a significant risk.<n>We propose an LLM-agentic framework, SE-VSim, to simulate SE attack mechanisms by generating multi-turn conversations.<n>We present a proof of concept, SE-OmniGuard, to offer personalized protection to users by leveraging prior knowledge of the victims personality.
  arXiv  Detail & Related papers  (2025-03-18T19:14:44Z)
• Can LLMs Simulate Social Media Engagement? A Study on Action-Guided Response Generation [51.44040615856536]
  This paper analyzes large language models' ability to simulate social media engagement through action guided response generation.<n>We benchmark GPT-4o-mini, O1-mini, and DeepSeek-R1 in social media engagement simulation regarding a major societal event.
  arXiv  Detail & Related papers  (2025-02-17T17:43:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.