Empirical Evaluation of Multi-Modal Touch Detection in Over-the-Shoulder Video Surveillance

• URL: http://arxiv.org/abs/2606.29504v1
• Date: Sun, 28 Jun 2026 17:05:56 GMT
• Title: Empirical Evaluation of Multi-Modal Touch Detection in Over-the-Shoulder Video Surveillance
• Abstract summary: We evaluate a framework designed to reconstruct keystroke events on mobile keypad interfaces from physical finger interactions.<n>We map relative touch coordinates to a reference screen layout to reconstruct typing sequences.<n>We conduct ablation, resolution decay, noise sensitivity, and proximity threshold tuning to characterize the system's operational envelope.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video Intelligence Surveillance (VIDINT) on over-the-shoulder footage is a proposed vector for monitoring human-computer interaction patterns without direct screen recording access. In this paper, we evaluate a Behavioral Intelligence (BEHINT) touch-detection framework designed to reconstruct keystroke events on mobile keypad interfaces from physical finger interactions. Our system integrates four parallel detection modalities: (1) anatomical hand landmarks via MediaPipe, (2) HSV skin color filtering, (3) temporal frame differencing for motion detection, and (4) shape-guided Canny edge analysis. We map relative touch coordinates to a reference screen layout to reconstruct typing sequences. Evaluation on a 120-frame first-person staged video of passcode entry reveals that while MediaPipe and Skin Detection fail to run autonomously due to partial hand occlusion and ambient noise, Motion-Only and Edge-Only configurations achieve F1-scores of 18.5% and 18.2%, respectively. The combined multi-modal configuration achieves an F1-score of 16.7% and a sequence similarity of 3.0% when mapped to the iOS passcode layout. We conduct ablation, resolution decay, noise sensitivity, and proximity threshold tuning to characterize the system's operational envelope. We then audit generalization on 5 real, publicly licensed third-person phone videos and find that the detector emits a median of 57 touch points per frame (peaking at 205), one to three orders of magnitude more than the rate of real taps, because the skin filter responds to the whole hand rather than to fingertip contact. The staged keystroke result does not survive contact with uncontrolled footage; the system does not achieve reliable keystroke reconstruction outside the calibrated staged setting.

Related papers

• Robust Slip Detection and Material Classification via Spatiotemporal Transformers on a Uniformly-Illuminated Visuo-Tactile Sensor [52.33204893988409]
  We propose a visuo-tactile sensor featuring customized uniform illumination, alongside a unified perception framework.<n>The proposed hardware-software framework provides high-fidelity feedback and a powerful multi-modal perception baseline for complex robotic manipulation.
  arXiv  Detail & Related papers  (2026-08-25T07:25:27Z)
• KineFuse: Kinematic-Aware Haptic Fusion for In-Hand Occluded-Object Pose Tracking [18.013999575547427]
  In-hand manipulation requires continuous 6D pose tracking, yet the manipulating fingers inevitably occlude the object from the camera.<n>We study how to structure the sparse haptic signals already available on multi-fingered hands, including proprioception, proximal force/torque, and binary contact.<n>We propose a kinematic-aware finger-level encoder and systematically compare it against four alternative designs through three levels of evaluation.
  arXiv  Detail & Related papers  (2026-07-16T11:07:47Z)
• Mobile UMI: Cross-View Diffusion Policy with Decoupled Kinematics for Mobile Manipulation [4.311650365696827]
  Recent wrist-mounted interfaces lower the cost of tabletop data collection, yet a single wrist view does not capture the global context required for base navigation.<n>generative policies introduce hundreds of milliseconds of inference latency, during which the base advances past predicted waypoints.<n>This paper presents Mobile UMI, a hardware-free demonstration framework that addresses both gaps through three components.
  arXiv  Detail & Related papers  (2026-05-20T08:33:53Z)
• Selfie-Capture Dynamics as an Auxiliary Signal Against Deepfakes and Injection Attacks for Mobile Identity Verification [4.2958409172092225]
  Mobile remote identity verification (RIdV) systems are exposed to attacks that manipulate or replace the facial video stream.<n>Recent European requirements, including ETSI TS 119 461 and CEN/TS 18099, motivate complementary evidence channels beyond camera-based presentation-attack detection.<n>This paper investigates whether passive motion traces recorded during selfie capture provide auxiliary evidence for spoof screening and user verification.
  arXiv  Detail & Related papers  (2026-04-30T20:46:18Z)
• QUACK! Making the (Rubber) Ducky Talk: A Systematic Study of Keystroke Dynamics for HID Injection Detection [60.105690843777666]
  We show that robust, privacy-preserving detection is achievable using lightweight models operating solely on timing features.<n>Our analysis reveals that attacker sophistication does not monotonically translate into improved evasion.
  arXiv  Detail & Related papers  (2026-04-17T08:53:24Z)
• Design and Validation of a Lightweight 1D CNN for Affective Touch Classification in Soft Plush Companions [59.49967357689445]
  Soft, sensorized companions offer a physically safe and emotionally intuitive interface for socially assistive technologies.<n>This study presents a complete open-source dilated real-based framework for the development and validation of deep learning models for affective touch recognition in soft interactive companions.
  arXiv  Detail & Related papers  (2026-04-16T10:20:05Z)
• Interpretable Human Activity Recognition for Subtle Robbery Detection in Surveillance Videos [0.0]
  Non-violent street robberies (snatch-and-run) are difficult to detect automatically because they are brief, subtle, and often indistinguishable from benign human interactions.<n>This paper presents a hybrid, pose-driven approach for detecting snatch-and-run events that combines real-time perception with an interpretable classification stage.
  arXiv  Detail & Related papers  (2026-04-15T18:37:28Z)
• COVTrack++: Learning Open-Vocabulary Multi-Object Tracking from Continuous Videos via a Synergistic Paradigm [59.26203051651017]
  C-TAO is the first continuously annotated training set for Open-Vocabulary Multi-Object Tracking (OVMOT)<n>For the framework bottleneck, we propose COVTrack++, a synergistic framework that achieves a bidirectional reciprocal mechanism between detection and association through three modules.<n>Experiments on TAO demonstrate state-of-the-art performance, with novel TETA reaching 35.4% and 30.5% on validation and test sets, improving novel AssocA by 4.8% and novel LocA by 5.8% over previous methods.
  arXiv  Detail & Related papers  (2026-03-25T07:20:27Z)
• OPENTOUCH: Bringing Full-Hand Touch to Real-World Interaction [93.88239833545623]
  We present OpenTouch, the first in-the-wild egocentric full-hand tactile dataset.<n>We show that tactile signals provide a compact yet powerful cue for grasp understanding.<n>We aim to advance multimodal egocentric perception, embodied learning, and contact-rich robotic manipulation.
  arXiv  Detail & Related papers  (2025-12-18T18:18:17Z)
• TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric Vision [25.271209425555906]
  We present a real-time pipeline that detects touch input from all ten fingers on any physical surface. Our method TouchInsight comprises a neural network to predict the moment of a touch event, the finger making contact, and the touch location.
  arXiv  Detail & Related papers  (2024-10-08T11:42:44Z)
• Mobile Behavioral Biometrics for Passive Authentication [65.94403066225384]
  This work carries out a comparative analysis of unimodal and multimodal behavioral biometric traits. Experiments are performed over HuMIdb, one of the largest and most comprehensive freely available mobile user interaction databases. In our experiments, the most discriminative background sensor is the magnetometer, whereas among touch tasks the best results are achieved with keystroke.
  arXiv  Detail & Related papers  (2022-03-14T17:05:59Z)
• HighlightMe: Detecting Highlights from Human-Centric Videos [52.84233165201391]
  We present a domain- and user-preference-agnostic approach to detect highlightable excerpts from human-centric videos. We use an autoencoder network equipped with spatial-temporal graph convolutions to detect human activities and interactions. We observe a 4-12% improvement in the mean average precision of matching the human-annotated highlights over state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-10-05T01:18:15Z)
• Activity Recognition with Moving Cameras and Few Training Examples: Applications for Detection of Autism-Related Headbanging [1.603589863010401]
  Activity recognition computer vision algorithms can be used to detect the presence of autism-related behaviors. We document the advantages and limitations of current feature representation techniques for activity recognition when applied to head banging detection. We create a computer vision classifier for detecting head banging in home videos using a time-distributed convolutional neural network.
  arXiv  Detail & Related papers  (2021-01-10T05:37:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.