Modular Deep Learning Framework for Assistive Perception: Gaze, Affect, and Speaker Identification

• URL: http://arxiv.org/abs/2511.20474v1
• Date: Tue, 25 Nov 2025 16:35:42 GMT
• Title: Modular Deep Learning Framework for Assistive Perception: Gaze, Affect, and Speaker Identification
• Authors: Akshit Pramod Anchan, Jewelith Thomas, Sritama Roy,
• Abstract summary: This research evaluates the feasibility of a modular architecture inspired by core functionalities of perceptive systems like 'Smart Eye'<n>We propose and benchmark three independent sensing modules: a Convolutional Neural Network (CNN) for eye state detection (drowsiness/attention), a deep CNN for facial expression recognition, and a Long Short-Term Memory (LSTM) network for voice-based speaker identification.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Developing comprehensive assistive technologies requires the seamless integration of visual and auditory perception. This research evaluates the feasibility of a modular architecture inspired by core functionalities of perceptive systems like 'Smart Eye.' We propose and benchmark three independent sensing modules: a Convolutional Neural Network (CNN) for eye state detection (drowsiness/attention), a deep CNN for facial expression recognition, and a Long Short-Term Memory (LSTM) network for voice-based speaker identification. Utilizing the Eyes Image, FER2013, and customized audio datasets, our models achieved accuracies of 93.0%, 97.8%, and 96.89%, respectively. This study demonstrates that lightweight, domain-specific models can achieve high fidelity on discrete tasks, establishing a validated foundation for future real-time, multimodal integration in resource-constrained assistive devices.

Related papers

• AVAR-Net: A Lightweight Audio-Visual Anomaly Recognition Framework with a Benchmark Dataset [11.179608136803447]
  Anomaly recognition plays a vital role in surveillance, transportation, healthcare and public safety.<n>Existing approaches rely solely on visual data, making them unreliable under challenging conditions.<n>The absence of large-scale synchronized audio-visual has hindered progress in multimodal anomaly recognition.
  arXiv  Detail & Related papers  (2025-10-15T14:56:00Z)
• Lightweight Joint Audio-Visual Deepfake Detection via Single-Stream Multi-Modal Learning Framework [19.53717894228692]
  Deepfakes are AI-synthesized multimedia data that may be abused for spreading misinformation.<n>We propose a lightweight network for audio-visual deepfake detection via a single-stream multi-modal learning framework.<n>Our method is significantly lightweight with only 0.48M parameters, yet it achieves superiority in both uni-modal and multi-modal deepfakes.
  arXiv  Detail & Related papers  (2025-06-09T02:13:04Z)
• ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models [68.46716645478661]
  Vision-language models (VLMs) have demonstrated remarkable capabilities in understanding and reasoning about visual content.<n>Current VLMs excel primarily at egocentric spatial reasoning (from the camera's perspective) but fail to generalize to allocentric viewpoints.<n>We introduce ViewSpatial-Bench, the first comprehensive benchmark designed specifically for multi-viewpoint spatial localization recognition evaluation.
  arXiv  Detail & Related papers  (2025-05-27T17:59:26Z)
• AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection [57.649223695021114]
  We present a novel weakly supervised framework that leverages audio-visual collaboration for robust video anomaly detection.<n>Our framework demonstrates superior performance across multiple benchmarks, with audio integration significantly boosting anomaly detection accuracy.
  arXiv  Detail & Related papers  (2025-04-06T13:59:16Z)
• LiDAR-based Object Detection with Real-time Voice Specifications [0.0]
  This paper presents a LiDAR-based object detection system with real-time voice specifications.<n>It achieves 87.0% validation accuracy on a 3000-sample subset, surpassing a 200-sample baseline of 67.5%.<n>A Tkinter prototype provides natural Indian male voice output using Edge TTS (en-IN-PrabhatNeural), alongside 3D visualizations and real-time feedback.
  arXiv  Detail & Related papers  (2025-04-03T16:50:38Z)
• Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision [83.0622534215881]
  This work proposes an industry-level omni-modal large language model (LLM) pipeline that integrates auditory, visual, and linguistic modalities.<n>Our pipeline consists of three main components: First, a modular framework enabling flexible configuration of various encoder-LLM-decoder architectures.<n>Second, a lightweight training strategy that pre-trains audio-language alignment on the state-of-the-art vision-language model Qwen2.5-VL.<n>Third, an audio synthesis pipeline that generates high-quality audio-text data from diverse real-world scenarios.
  arXiv  Detail & Related papers  (2025-02-26T17:26:36Z)
• Integrating Object Detection Modality into Visual Language Model for Enhanced Autonomous Driving Agent [8.212818176634116]
  We extend the Llama-Adapter architecture by incorporating a YOLOS-based detection network alongside the CLIP perception network. Our approach introduces camera ID-separators to improve multi-view processing, crucial for comprehensive environmental awareness.
  arXiv  Detail & Related papers  (2024-11-08T15:50:30Z)
• Query-by-Example Keyword Spotting Using Spectral-Temporal Graph Attentive Pooling and Multi-Task Learning [11.182456667123835]
  We present a novel Query-by-Example (QbyE) KWS system that employs spectral-temporal attentive graph pooling and multi-task learning. This framework aims to effectively learn speaker-invariant and linguistic-informative embeddings for QbyE KWS tasks.
  arXiv  Detail & Related papers  (2024-08-27T03:44:57Z)
• NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals [58.83169560132308]
  We introduce NNsight and NDIF, technologies that work in tandem to enable scientific study of the representations and computations learned by very large neural networks.
  arXiv  Detail & Related papers  (2024-07-18T17:59:01Z)
• AGO-Net: Association-Guided 3D Point Cloud Object Detection Network [86.10213302724085]
  We propose a novel 3D detection framework that associates intact features for objects via domain adaptation. We achieve new state-of-the-art performance on the KITTI 3D detection benchmark in both accuracy and speed.
  arXiv  Detail & Related papers  (2022-08-24T16:54:38Z)
• AutoSpeech: Neural Architecture Search for Speaker Recognition [108.69505815793028]
  We propose the first neural architecture search approach approach for the speaker recognition tasks, named as AutoSpeech. Our algorithm first identifies the optimal operation combination in a neural cell and then derives a CNN model by stacking the neural cell for multiple times. Results demonstrate that the derived CNN architectures significantly outperform current speaker recognition systems based on VGG-M, ResNet-18, and ResNet-34 back-bones, while enjoying lower model complexity.
  arXiv  Detail & Related papers  (2020-05-07T02:53:47Z)
• Deep Speaker Embeddings for Far-Field Speaker Recognition on Short Utterances [53.063441357826484]
  Speaker recognition systems based on deep speaker embeddings have achieved significant performance in controlled conditions. Speaker verification on short utterances in uncontrolled noisy environment conditions is one of the most challenging and highly demanded tasks. This paper presents approaches aimed to achieve two goals: a) improve the quality of far-field speaker verification systems in the presence of environmental noise, reverberation and b) reduce the system qualitydegradation for short utterances.
  arXiv  Detail & Related papers  (2020-02-14T13:34:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.