Multi-Modal Machine Learning for Early Trust Prediction in Human-AI Interaction Using Face Image and GSR Bio Signals

• URL: http://arxiv.org/abs/2511.21908v2
• Date: Wed, 03 Dec 2025 23:35:04 GMT
• Title: Multi-Modal Machine Learning for Early Trust Prediction in Human-AI Interaction Using Face Image and GSR Bio Signals
• Abstract summary: This study proposes a multi-modal machine learning framework that combines image and galvanic skin response (GSR) data to predict early user trust in AI- or human-generated recommendations in a simulated ADHD mHealth context.<n>Results showed that combining facial and physiological cues significantly improved prediction performance.
• Score: 0.8594140167290097
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Predicting human trust in AI systems is crucial for safe integration of AI-based decision support tools, especially in healthcare. This study proposes a multi-modal machine learning framework that combines image and galvanic skin response (GSR) data to predict early user trust in AI- or human-generated recommendations in a simulated ADHD mHealth context. Facial video data were processed using OpenCV for frame extraction and transferred learning with a pre-trained transformer model to derive emotional features. Concurrently, GSR signals were decomposed into tonic and phasic components to capture physiological arousal patterns. Two temporal windows were defined for trust prediction: the Early Detection Window (6 to 3 seconds before decision-making) and the Proximal Detection Window (3 to 0 seconds before decision-making). For each window, trust prediction was conducted separately using image-based, GSR-based, and multimodal (image + GSR) features. Each modality was analyzed using machine learning algorithms, and the top-performing unimodal models were integrated through a multimodal stacking ensemble for final prediction. Experimental results showed that combining facial and physiological cues significantly improved prediction performance. The multimodal stacking framework achieved an accuracy of 0.83, F1-score of 0.88, and ROC-AUC of 0.87 in the Early Detection Window, and an accuracy of 0.75, F1-score of 0.82, and ROC-AUC of 0.66 in the Proximal Detection Window. These results demonstrate the potential of bio signals as real-time, objective markers of user trust, enabling adaptive AI systems that dynamically adjust their responses to maintain calibrated trust which is a critical capability in mental health applications where mis-calibrated trust can affect diagnostic and treatment outcomes.

Related papers

• EGRL: Edge generation-guided relation-aware learning for RNA-protein interaction prediction [41.17218504310278]
  We propose Edge Generation-guided Relation-aware Learning (EGRL)<n>EGRL is a novel framework with several key components.<n> Comprehensive evaluations on four benchmark datasets demonstrate that EGRL achieves competitive overall performance.
  arXiv  Detail & Related papers  (2026-08-13T07:47:44Z)
• Perception, Verdict, and Evolution: Hindsight-Driven Self-Refining Forensics Agent for AI-Generated Image Detection [80.15317858033534]
  ForeAgent is an agentic forensics framework for AI-generated image detection with iterative self-evolution.<n>ForeAgent achieves state-of-the-art performance on the Chameleon benchmark, reaching 82.18% accuracy.<n>ForeAgent produces more consistent and causally grounded reasoning compared to GPT-5 and GPT-5-mini.
  arXiv  Detail & Related papers  (2026-06-25T02:59:33Z)
• A Two-Stage, Object-Centric Deep Learning Framework for Robust Exam Cheating Detection [0.0]
  Academic integrity continues to face the persistent challenge of examination cheating.<n>Traditional invigilation relies on human observation, which is inefficient, costly, and prone to errors at scale.<n>We propose an improvement over a simple two-stage framework for exam cheating detection using well-known technologies.
  arXiv  Detail & Related papers  (2026-04-17T16:53:08Z)
• Design and Validation of a Lightweight 1D CNN for Affective Touch Classification in Soft Plush Companions [59.49967357689445]
  Soft, sensorized companions offer a physically safe and emotionally intuitive interface for socially assistive technologies.<n>This study presents a complete open-source dilated real-based framework for the development and validation of deep learning models for affective touch recognition in soft interactive companions.
  arXiv  Detail & Related papers  (2026-04-16T10:20:05Z)
• Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment [56.62016795093786]
  Cerebra is an interactive multi-agent AI team that coordinates specialized agents for EHR, clinical notes, and medical imaging analysis.<n>It supports privacy-preserving deployment by operating on structured representations and remains robust when modalities are incomplete.<n>Cerebra significantly improved expert performance, increasing accuracy by 17.5 percentage points in prospective dementia risk estimation.
  arXiv  Detail & Related papers  (2026-03-23T05:46:45Z)
• Transparent Early ICU Mortality Prediction with Clinical Transformer and Per-Case Modality Attribution [42.85462513661566]
  We present a lightweight, transparent multimodal ensemble that fuses physiological time-series measurements with unstructured clinical notes from the first 48 hours of an ICU stay.<n>A logistic regression model combines predictions from two modality-specific models: a bidirectional LSTM for vitals and a finetuned ClinicalModernBERT transformer for notes.<n>On the MIMIC-III benchmark, our late-fusion ensemble improves discrimination over the best single model while maintaining well-calibrated predictions.
  arXiv  Detail & Related papers  (2025-11-19T20:11:49Z)
• ActiTect: A Generalizable Machine Learning Pipeline for REM Sleep Behavior Disorder Screening through Standardized Actigraphy [1.2106870940376342]
  Isolated rapid eye movement sleep behavior disorder (iRBD) is a major prodromal marker of $$-synucleinopathies.<n> wrist-worn actimeters hold significant potential for detecting RBD in large-scale screening efforts.<n>This study presents ActiTect, a fully automated, open-source machine learning tool to identify RBD from actigraphy recordings.
  arXiv  Detail & Related papers  (2025-11-07T13:18:20Z)
• A Hybrid Deep Learning and Forensic Approach for Robust Deepfake Detection [0.0]
  Existing deepfake detection methods either rely on deep learning, which suffers from poor generalization and vulnerability to distortions, or forensic analysis, which is interpretable but limited against new manipulation techniques.<n>This study proposes a hybrid framework that fuses forensic features, including noise residuals, JPEG compression traces, and frequency-domain descriptors, with deep learning representations from CNNs and vision transformers.
  arXiv  Detail & Related papers  (2025-10-31T11:32:52Z)
• Predicting Diabetic Retinopathy Using a Two-Level Ensemble Model [0.6445605125467574]
  Diabetic retinopathy is a leading cause of blindness in working-age adults.<n>Image-based AI tools have shown limitations in early-stage detection.<n>We propose a non-image-based, two-level ensemble model for DR prediction using routine laboratory test results.
  arXiv  Detail & Related papers  (2025-10-01T16:19:57Z)
• Automatic Cough Analysis for Non-Small Cell Lung Cancer Detection [33.37223681850477]
  Early detection of non-small cell lung cancer (NSCLC) is critical for improving patient outcomes.<n>We explore the use of automatic cough analysis as a pre-screening tool for distinguishing between NSCLC patients and healthy controls.<n>Recordings were analyzed using machine learning techniques, such as support vector machine (SVM) and XGBoost.
  arXiv  Detail & Related papers  (2025-07-25T11:30:22Z)
• Implementing Trust in Non-Small Cell Lung Cancer Diagnosis with a Conformalized Uncertainty-Aware AI Framework in Whole-Slide Images [37.3701890138561]
  TRUECAM is a framework designed to ensure both data and model trustworthiness in non-small cell lung cancer subtyping with whole-slide images.<n>An AI model wrapped with TRUECAM significantly outperforms models that lack such guidance, in terms of classification accuracy, robustness, interpretability, and data efficiency.
  arXiv  Detail & Related papers  (2024-12-28T02:22:47Z)
• MANGO: Multimodal Acuity traNsformer for intelliGent ICU Outcomes [11.385654412265461]
  We present MANGO: the Multimodal Acuity traNsformer for intelliGent ICU outcomes.<n>It is designed to enhance the prediction of patient acuity states, transitions, and the need for life-sustaining therapy.
  arXiv  Detail & Related papers  (2024-12-13T23:51:15Z)
• Learning a Neural Association Network for Self-supervised Multi-Object Tracking [39.65022597490719]
  This paper introduces a novel framework to learn data association for multi-object tracking in a self-supervised manner.<n>We present a novel expectation (EM) algorithm that trains a neural network to associate detections for tracking.<n>We evaluate our approach on the challenging MOT17, MOT20, and BDD100K datasets and achieve state-of-the-art results.
  arXiv  Detail & Related papers  (2024-11-18T12:22:29Z)
• Machine Learning for ALSFRS-R Score Prediction: Making Sense of the Sensor Data [44.99833362998488]
  Amyotrophic Lateral Sclerosis (ALS) is a rapidly progressive neurodegenerative disease that presents individuals with limited treatment options. The present investigation, spearheaded by the iDPP@CLEF 2024 challenge, focuses on utilizing sensor-derived data obtained through an app.
  arXiv  Detail & Related papers  (2024-07-10T19:17:23Z)
• A Progressive Risk Formulation for Enhanced Deep Learning based Total Knee Replacement Prediction in Knee Osteoarthritis [36.12790384412525]
  We developed deep learning models for predicting Total Knee Replacement (TKR) need within various time horizons in knee osteoarthritis patients.<n>The models can perform TKR prediction using a single scan, and furthermore when a previous scan is available, they leverage a progressive risk formulation to improve their predictions.<n>Knee radiographs and MRIs from the Osteoarthritis Initiative (OAI) and Multicenter Osteoarthritis Study (MOST) were used in this work.
  arXiv  Detail & Related papers  (2024-06-14T15:24:49Z)
• Comprehensive Multimodal Deep Learning Survival Prediction Enabled by a Transformer Architecture: A Multicenter Study in Glioblastoma [4.578027879885667]
  This research aims to improve glioblastoma survival prediction by integrating MR images, clinical and molecular-pathologic data in a transformer-based deep learning model. The model employs self-supervised learning techniques to effectively encode the high-dimensional MRI input for integration with non-imaging data using cross-attention.
  arXiv  Detail & Related papers  (2024-05-21T17:44:48Z)
• SQUWA: Signal Quality Aware DNN Architecture for Enhanced Accuracy in Atrial Fibrillation Detection from Noisy PPG Signals [37.788535094404644]
  Atrial fibrillation (AF) significantly increases the risk of stroke, heart disease, and mortality. Photoplethysmography ( PPG) signals are susceptible to corruption from motion artifacts and other factors often encountered in ambulatory settings. We propose a novel deep learning model, designed to learn how to retain accurate predictions from partially corrupted PPG.
  arXiv  Detail & Related papers  (2024-04-15T01:07:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.