SEAR: A Multimodal Dataset for Analyzing AR-LLM-Driven Social Engineering Behaviors

• URL: http://arxiv.org/abs/2505.24458v1
• Date: Fri, 30 May 2025 10:46:13 GMT
• Title: SEAR: A Multimodal Dataset for Analyzing AR-LLM-Driven Social Engineering Behaviors
• Abstract summary: The SEAR dataset is a novel multimodal resource designed to study the emerging threat of social engineering (SE) attacks orchestrated through augmented reality (AR) and multimodal large language models (LLMs)<n>This dataset captures 180 annotated conversations across 60 participants in simulated adversarial scenarios.<n>It comprises synchronized AR-captured visual/audio cues (e.g., facial expressions, vocal tones), environmental context, and curated social media profiles, alongside subjective metrics such as trust ratings and susceptibility assessments.
• Score: 8.285642026459179
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The SEAR Dataset is a novel multimodal resource designed to study the emerging threat of social engineering (SE) attacks orchestrated through augmented reality (AR) and multimodal large language models (LLMs). This dataset captures 180 annotated conversations across 60 participants in simulated adversarial scenarios, including meetings, classes and networking events. It comprises synchronized AR-captured visual/audio cues (e.g., facial expressions, vocal tones), environmental context, and curated social media profiles, alongside subjective metrics such as trust ratings and susceptibility assessments. Key findings reveal SEAR's alarming efficacy in eliciting compliance (e.g., 93.3% phishing link clicks, 85% call acceptance) and hijacking trust (76.7% post-interaction trust surge). The dataset supports research in detecting AR-driven SE attacks, designing defensive frameworks, and understanding multimodal adversarial manipulation. Rigorous ethical safeguards, including anonymization and IRB compliance, ensure responsible use. The SEAR dataset is available at https://github.com/INSLabCN/SEAR-Dataset.

Related papers

• MODF-SIR: A Multi-agent Omni-modal Distilled Framework for Social Intelligence Reasoning [80.41987492855971]
  We propose a multi-agent collaborative framework built upon a lightweight Multimodal Large Language Model (MLLM)<n>Within this architecture, multi-modal data pertinent to social intelligence is precisely localized.<n>With around 30% of training data from IntentTrain, we achieve state-of-the-art results.
  arXiv  Detail & Related papers  (2026-06-10T12:44:02Z)
• A Synthetic Conversational Smishing Dataset for Social Engineering Detection [0.0]
  This paper presents a dataset of 3,201 labeled multi-round conversations designed to emulate realistic conversational smishing attacks.<n>Results show that TF-IDF-based models consistently outperform those using engineered features alone.
  arXiv  Detail & Related papers  (2026-04-13T17:26:17Z)
• AgentIR: Reasoning-Aware Retrieval for Deep Research Agents [76.29382561831105]
  Deep Research agents generate explicit natural language reasoning before each search call.<n> Reasoning-Aware Retrieval embeds the agent's reasoning trace alongside its query.<n>DR- Synth generates Deep Research retriever training data from standard QA datasets.<n>AgentIR-4B achieves 68% accuracy with the open-weight agent Tongyi-DeepResearch.
  arXiv  Detail & Related papers  (2026-03-04T18:47:26Z)
• Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition [71.5328300638085]
  Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions.<n>Existing methods, including two-stage methods, tightly couple interaction recognition with a specific detector.<n>We propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR.
  arXiv  Detail & Related papers  (2026-02-16T19:01:31Z)
• S-DAPT-2026: A Stage-Aware Synthetic Dataset for Advanced Persistent Threat Detection [0.0538441598991272]
  This paper presents a near realistic synthetic APT dataset and an efficient alert correlation framework.<n>The proposed approach introduces a machine learning based correlation module that employs K Nearest Neighbors (KNN) clustering with a cosine similarity metric to group semantically related alerts.<n>A comprehensive statistical characterization of the dataset is provided to facilitate aware and support APT stage predictions.
  arXiv  Detail & Related papers  (2026-01-10T21:25:41Z)
• Noise-Aware and Dynamically Adaptive Federated Defense Framework for SAR Image Target Recognition [41.71964972109509]
  Federated learning (FL) provides an emerging computational intelligence paradigm for SAR image target recognition.<n>FL confronts critical security risks, where malicious clients can exploit SAR's multiplicative speckle noise to conceal backdoor triggers.<n>We propose NADAFD, a noise-aware and dynamically adaptive federated defense framework that integrates frequency-domain, spatial-domain, and client-behavior analyses.
  arXiv  Detail & Related papers  (2025-12-31T17:24:15Z)
• Towards Unified Multimodal Misinformation Detection in Social Media: A Benchmark Dataset and Baseline [56.790045049514326]
  Two major forms of deception dominate: human-crafted misinformation and AI-generated content.<n>We propose Unified Multimodal Fake Content Detection (UMFDet), a framework designed to handle both forms of deception.<n>UMFDet achieves robust and consistent performance across both misinformation types, outperforming specialized baselines.
  arXiv  Detail & Related papers  (2025-09-30T09:26:32Z)
• On the Feasibility of Using MultiModal LLMs to Execute AR Social Engineering Attacks [8.28564202645918]
  We propose a framework for orchestrating AR-driven Social Engineering attacks using Multimodal Large Language Models.<n>Our results show that SEAR is highly effective at eliciting high-risk behaviors.<n>We identify notable limitations such as occasionally artificial'' due to perceived authenticity gaps.
  arXiv  Detail & Related papers  (2025-04-16T05:18:36Z)
• Composed Multi-modal Retrieval: A Survey of Approaches and Applications [81.54640206021757]
  Composed Multi-modal Retrieval (CMR) emerges as a pivotal next-generation technology.<n>CMR enables users to query images or videos by integrating a reference visual input with textual modifications.<n>This paper provides a comprehensive survey of CMR, covering its fundamental challenges, technical advancements, and applications.
  arXiv  Detail & Related papers  (2025-03-03T09:18:43Z)
• CBW: Towards Dataset Ownership Verification for Speaker Verification via Clustering-based Backdoor Watermarking [85.68235482145091]
  Large-scale speech datasets have become valuable intellectual property.<n>We propose a novel dataset ownership verification method.<n>Our approach introduces a clustering-based backdoor watermark (CBW)<n>We conduct extensive experiments on benchmark datasets, verifying the effectiveness and robustness of our method against potential adaptive attacks.
  arXiv  Detail & Related papers  (2025-03-02T02:02:57Z)
• Mind the Gap! Static and Interactive Evaluations of Large Audio Models [55.87220295533817]
  Large Audio Models (LAMs) are designed to power voice-native experiences.<n>This study introduces an interactive approach to evaluate LAMs and collect 7,500 LAM interactions from 484 participants.
  arXiv  Detail & Related papers  (2025-02-21T20:29:02Z)
• Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering [25.577314828249897]
  We propose a novel dataset, MUSIC-AVQA-R, crafted in two steps: rephrasing questions within the test split of a public dataset (MUSIC-AVQA) and introducing distribution shifts to split questions.<n> Experimental results show that this architecture achieves state-of-the-art performance on MUSIC-AVQA-R, notably obtaining a significant improvement of 9.32%.
  arXiv  Detail & Related papers  (2024-04-18T09:16:02Z)
• AV-RIR: Audio-Visual Room Impulse Response Estimation [49.469389715876915]
  Accurate estimation of Room Impulse Response (RIR) is important for speech processing and AR/VR applications. We propose AV-RIR, a novel multi-modal multi-task learning approach to accurately estimate the RIR from a given reverberant speech signal and visual cues of its corresponding environment.
  arXiv  Detail & Related papers  (2023-11-30T22:58:30Z)
• rWISDM: Repaired WISDM, a Public Dataset for Human Activity Recognition [0.0]
  Human Activity Recognition (HAR) has become a spotlight in recent scientific research because of its applications in various domains such as healthcare, athletic competitions, smart cities, and smart home. This paper presents the methods by which other researchers may identify and correct similar problems in public datasets.
  arXiv  Detail & Related papers  (2023-05-17T13:55:50Z)
• Jointly Learning Visual and Auditory Speech Representations from Raw Data [108.68531445641769]
  RAVEn is a self-supervised multi-modal approach to jointly learn visual and auditory speech representations. Our design is asymmetric w.r.t. driven by the inherent differences between video and audio. RAVEn surpasses all self-supervised methods on visual speech recognition.
  arXiv  Detail & Related papers  (2022-12-12T21:04:06Z)
• PS-ARM: An End-to-End Attention-aware Relation Mixer Network for Person Search [56.02761592710612]
  We propose a novel attention-aware relation mixer (ARM) for module person search. Our ARM module is native and does not rely on fine-grained supervision or topological assumptions. Our PS-ARM achieves state-of-the-art performance on both datasets.
  arXiv  Detail & Related papers  (2022-10-07T10:04:12Z)
• EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environments [43.05826988957987]
  We release a dataset that contains over 5 hours of multi-modal data useful for training and testing algorithms for the application of improving conversations for an AR glasses wearer. We provide speech intelligibility, quality and signal-to-noise ratio improvement results for a baseline method and show improvements across all tested metrics.
  arXiv  Detail & Related papers  (2021-07-09T02:00:47Z)
• ROSbag-based Multimodal Affective Dataset for Emotional and Cognitive States [0.9786690381850356]
  This paper introduces a new ROSbag-based multimodal affective dataset for emotional and cognitive states generated using Robot Operating System (ROS) We utilized images and sounds from the International Affective Pictures System (IAPS) and the International Affective Digitized Sounds (IADS) to stimulate targeted emotions. The generated affective dataset consists of 1,602 ROSbag files, and size of the dataset is about 787GB.
  arXiv  Detail & Related papers  (2020-06-09T08:09:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.