SEAR: A Multimodal Dataset for Analyzing AR-LLM-Driven Social Engineering Behaviors

• URL: http://arxiv.org/abs/2505.24458v1
• Date: Fri, 30 May 2025 10:46:13 GMT
• Title: SEAR: A Multimodal Dataset for Analyzing AR-LLM-Driven Social Engineering Behaviors
• Authors: Tianlong Yu, Chenghang Ye, Zheyu Yang, Ziyi Zhou, Cui Tang, Zui Tao, Jun Zhang, Kailong Wang, Liting Zhou, Yang Yang, Ting Bi,
• Abstract summary: The SEAR dataset is a novel multimodal resource designed to study the emerging threat of social engineering (SE) attacks orchestrated through augmented reality (AR) and multimodal large language models (LLMs)<n>This dataset captures 180 annotated conversations across 60 participants in simulated adversarial scenarios.<n>It comprises synchronized AR-captured visual/audio cues (e.g., facial expressions, vocal tones), environmental context, and curated social media profiles, alongside subjective metrics such as trust ratings and susceptibility assessments.
• Score: 8.285642026459179
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The SEAR Dataset is a novel multimodal resource designed to study the emerging threat of social engineering (SE) attacks orchestrated through augmented reality (AR) and multimodal large language models (LLMs). This dataset captures 180 annotated conversations across 60 participants in simulated adversarial scenarios, including meetings, classes and networking events. It comprises synchronized AR-captured visual/audio cues (e.g., facial expressions, vocal tones), environmental context, and curated social media profiles, alongside subjective metrics such as trust ratings and susceptibility assessments. Key findings reveal SEAR's alarming efficacy in eliciting compliance (e.g., 93.3% phishing link clicks, 85% call acceptance) and hijacking trust (76.7% post-interaction trust surge). The dataset supports research in detecting AR-driven SE attacks, designing defensive frameworks, and understanding multimodal adversarial manipulation. Rigorous ethical safeguards, including anonymization and IRB compliance, ensure responsible use. The SEAR dataset is available at https://github.com/INSLabCN/SEAR-Dataset.

Related papers

• On the Feasibility of Using MultiModal LLMs to Execute AR Social Engineering Attacks [8.28564202645918]
  We propose a framework for orchestrating AR-driven Social Engineering attacks using Multimodal Large Language Models.<n>Our results show that SEAR is highly effective at eliciting high-risk behaviors.<n>We identify notable limitations such as occasionally artificial'' due to perceived authenticity gaps.
  arXiv  Detail & Related papers  (2025-04-16T05:18:36Z)
• Composed Multi-modal Retrieval: A Survey of Approaches and Applications [81.54640206021757]
  Composed Multi-modal Retrieval (CMR) emerges as a pivotal next-generation technology.<n>CMR enables users to query images or videos by integrating a reference visual input with textual modifications.<n>This paper provides a comprehensive survey of CMR, covering its fundamental challenges, technical advancements, and applications.
  arXiv  Detail & Related papers  (2025-03-03T09:18:43Z)
• CBW: Towards Dataset Ownership Verification for Speaker Verification via Clustering-based Backdoor Watermarking [85.68235482145091]
  Large-scale speech datasets have become valuable intellectual property.<n>We propose a novel dataset ownership verification method.<n>Our approach introduces a clustering-based backdoor watermark (CBW)<n>We conduct extensive experiments on benchmark datasets, verifying the effectiveness and robustness of our method against potential adaptive attacks.
  arXiv  Detail & Related papers  (2025-03-02T02:02:57Z)
• Mind the Gap! Static and Interactive Evaluations of Large Audio Models [55.87220295533817]
  Large Audio Models (LAMs) are designed to power voice-native experiences.<n>This study introduces an interactive approach to evaluate LAMs and collect 7,500 LAM interactions from 484 participants.
  arXiv  Detail & Related papers  (2025-02-21T20:29:02Z)
• Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering [25.577314828249897]
  We propose a novel dataset, MUSIC-AVQA-R, crafted in two steps: rephrasing questions within the test split of a public dataset (MUSIC-AVQA) and introducing distribution shifts to split questions.<n> Experimental results show that this architecture achieves state-of-the-art performance on MUSIC-AVQA-R, notably obtaining a significant improvement of 9.32%.
  arXiv  Detail & Related papers  (2024-04-18T09:16:02Z)
• AV-RIR: Audio-Visual Room Impulse Response Estimation [49.469389715876915]
  Accurate estimation of Room Impulse Response (RIR) is important for speech processing and AR/VR applications. We propose AV-RIR, a novel multi-modal multi-task learning approach to accurately estimate the RIR from a given reverberant speech signal and visual cues of its corresponding environment.
  arXiv  Detail & Related papers  (2023-11-30T22:58:30Z)
• rWISDM: Repaired WISDM, a Public Dataset for Human Activity Recognition [0.0]
  Human Activity Recognition (HAR) has become a spotlight in recent scientific research because of its applications in various domains such as healthcare, athletic competitions, smart cities, and smart home. This paper presents the methods by which other researchers may identify and correct similar problems in public datasets.
  arXiv  Detail & Related papers  (2023-05-17T13:55:50Z)
• Jointly Learning Visual and Auditory Speech Representations from Raw Data [108.68531445641769]
  RAVEn is a self-supervised multi-modal approach to jointly learn visual and auditory speech representations. Our design is asymmetric w.r.t. driven by the inherent differences between video and audio. RAVEn surpasses all self-supervised methods on visual speech recognition.
  arXiv  Detail & Related papers  (2022-12-12T21:04:06Z)
• PS-ARM: An End-to-End Attention-aware Relation Mixer Network for Person Search [56.02761592710612]
  We propose a novel attention-aware relation mixer (ARM) for module person search. Our ARM module is native and does not rely on fine-grained supervision or topological assumptions. Our PS-ARM achieves state-of-the-art performance on both datasets.
  arXiv  Detail & Related papers  (2022-10-07T10:04:12Z)
• EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environments [43.05826988957987]
  We release a dataset that contains over 5 hours of multi-modal data useful for training and testing algorithms for the application of improving conversations for an AR glasses wearer. We provide speech intelligibility, quality and signal-to-noise ratio improvement results for a baseline method and show improvements across all tested metrics.
  arXiv  Detail & Related papers  (2021-07-09T02:00:47Z)
• ROSbag-based Multimodal Affective Dataset for Emotional and Cognitive States [0.9786690381850356]
  This paper introduces a new ROSbag-based multimodal affective dataset for emotional and cognitive states generated using Robot Operating System (ROS) We utilized images and sounds from the International Affective Pictures System (IAPS) and the International Affective Digitized Sounds (IADS) to stimulate targeted emotions. The generated affective dataset consists of 1,602 ROSbag files, and size of the dataset is about 787GB.
  arXiv  Detail & Related papers  (2020-06-09T08:09:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.