Frozen Multimodal Embeddings for AI-Assisted Interview Assessment of Personality and Cognitive Ability

• URL: http://arxiv.org/abs/2606.11930v2
• Date: Thu, 11 Jun 2026 09:34:52 GMT
• Title: Frozen Multimodal Embeddings for AI-Assisted Interview Assessment of Personality and Cognitive Ability
• Abstract summary: This paper presents our solution for the ACM Multimedia AVI Challenge 2026.<n>Track1 predicts self-reported HEXACO personality traits from personality-related interview responses, and Track2 classifies cognitive ability levels.<n>We use frozen multimodal encoders, including CLIP for visual features, Whisper for acoustic features and transcripts, and RoBERTa, E5, and DeBERTaV3 for textual representations.
• Score: 0.20999222360659606
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Predicting psychological traits from asynchronous video interviews (AVIs) is a challenging problem in AI-assisted interview assessment because labeled datasets are limited while each response contains high-dimensional visual, acoustic, and verbal signals. This paper presents our solution for the ACM Multimedia AVI Challenge 2026, which evaluates two tasks: Track~1 predicts self-reported HEXACO personality traits from personality-related interview responses, and Track~2 classifies cognitive ability levels from structured AVI responses. We treat the problem as a small-sample representation learning task. Instead of fine-tuning large pretrained models, we use frozen multimodal encoders, including CLIP for visual features, Whisper for acoustic features and transcripts, and RoBERTa, E5, and DeBERTaV3 for textual representations, followed by low-capacity downstream models. For Track~1, our trait-specific regression and late-fusion system achieves an average validation MSE of 0.2696, improving over the official baseline of 0.3334. Ablation results show a three-step improvement from a global model (0.3189), to per-trait modeling (0.2871), to per-trait late fusion (0.2696), corresponding to a 19.1% relative MSE reduction over the official baseline. For Track~2, a compact subject-attribute baseline reaches 0.5781 accuracy, while our multimodal ensemble reaches 0.5313, both above the official baseline of 0.4062. We interpret this result as evidence of possible subject-attribute shortcuts in the validation split rather than robust cognitive inference from AVI content. Overall, our findings suggest that AVI-based psychological assessment benefits from trait-specific multimodal modeling, but cognitive ability prediction requires careful control of dataset shortcuts.

Related papers

• Parameter-efficient Prompt Tuning of Vision Foundation Model With Adaptive Focal Loss for Interpretable MCI Screening [1.0195618602298684]
  We propose a parameter-efficient framework utilizing frozen DINOv2-adapted model adapted via three modality-specific learnable prompt tokens.<n>The proposed architecture yields an MCI-class F1 of 0.641 and an AUC of 0.795, outperforming the computationally heavier ResViT baseline by 0.110 in MCI-class F1.
  arXiv  Detail & Related papers  (2026-07-16T14:26:22Z)
• Multimodal Evaluator Preference Collapse: Cross-Modal Contagion in Self-Evolving Agents [1.0152838128195467]
  We show that Evaluator Preference Collapse ( EPC) is dramatically amplified in multimodal settings.<n>We then demonstrate a novel phenomenon we term cross-modal contagion.<n>We identify cross-model evaluator architecture as the primary risk factor for preference drift.
  arXiv  Detail & Related papers  (2026-06-15T13:18:20Z)
• Traits Run Deeper: Trait-Specific Asymmetric Fusion for Personality Assessment [53.50824432699408]
  Traits Run Deeper is a novel personality assessment framework.<n>It consists of three components: the Multimodal Foundation Representation (MFR), the Trait-Specific Modality Fusion (TSMF) and the Distribution-Calibrated Personality Regression (DCPR)
  arXiv  Detail & Related papers  (2026-06-09T06:38:36Z)
• Design and Validation of a Lightweight 1D CNN for Affective Touch Classification in Soft Plush Companions [59.49967357689445]
  Soft, sensorized companions offer a physically safe and emotionally intuitive interface for socially assistive technologies.<n>This study presents a complete open-source dilated real-based framework for the development and validation of deep learning models for affective touch recognition in soft interactive companions.
  arXiv  Detail & Related papers  (2026-04-16T10:20:05Z)
• MCQ Difficulty Prediction via Modeling Learner Heterogeneity Using Data-Driven Cognitive Profiling [0.11458853556386796]
  We propose a persona-driven framework that replaces theoretical ability sampling with data-driven cognitive profiling.<n>We identify behavioral personas via latent class analysis (LCA), then condition a large language model (LLM) to simulate response distributions for each persona.<n>The discovered personas are interpretable and offer insights into why items are difficult, with potential applications to diagnostic assessment design.
  arXiv  Detail & Related papers  (2026-04-13T19:18:14Z)
• AdaptStress: Online Adaptive Learning for Interpretable and Personalized Stress Prediction Using Multivariate and Sparse Physiological Signals [1.593065406609169]
  This paper presents a novel, explainable, and individualized approach for stress prediction using physiological data from consumer-grade smartwatches.<n>We develop a time series forecasting model that leverages multivariate features, including heart rate variability, activity patterns, and sleep metrics, to predict stress levels.<n>Our model achieved performance with an MSE of 0.053, MAE of 0.190, and RMSE of 0.226 in optimal settings.
  arXiv  Detail & Related papers  (2026-02-19T20:57:35Z)
• Listening to the Unspoken: Exploring "365" Aspects of Multimodal Interview Performance Assessment [45.92718704785823]
  We propose a novel and comprehensive framework that explores 365'' aspects of interview performance.<n>The framework employs modality-specific feature extractors to encode heterogeneous data streams.<n>By listening to the unspoken, our approach captures both explicit and implicit cues from multimodal data.
  arXiv  Detail & Related papers  (2025-07-30T13:37:06Z)
• Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations? [37.703287009808896]
  Finetuning can cause spurious correlations to arise between non-essential features and the target labels.<n>We develop a benchmark by sourcing GPT-4o errors on real-world visual-question-answering (VQA) benchmarks.<n>We evaluate 15 open and closed-source LVLMs on SpuriVerse, finding that even state-of-the-art closed-source models struggle significantly.
  arXiv  Detail & Related papers  (2025-06-23T06:11:43Z)
• ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models [48.99485386990197]
  We conduct a case study using a synthetic dataset solvable only through visual reasoning.<n>We then introduce ChartMuseum, a new Chart Question Answering (QA) benchmark containing 1,162 expert-annotated questions.<n>Although humans achieve 93% accuracy, the best-performing model Gemini-2.5-Pro attains only 63.0%, and the leading open-source LVLM Qwen2.5-VL-72B-Instruct achieves only 38.5%.
  arXiv  Detail & Related papers  (2025-05-19T17:59:27Z)
• Mind the Gap! Static and Interactive Evaluations of Large Audio Models [55.87220295533817]
  Large Audio Models (LAMs) are designed to power voice-native experiences.<n>This study introduces an interactive approach to evaluate LAMs and collect 7,500 LAM interactions from 484 participants.
  arXiv  Detail & Related papers  (2025-02-21T20:29:02Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• Improving Selective Visual Question Answering by Learning from Your Peers [74.20167944693424]
  Visual Question Answering (VQA) models can have difficulties abstaining from answering when they are wrong. We propose Learning from Your Peers (LYP) approach for training multimodal selection functions for making abstention decisions. Our approach uses predictions from models trained on distinct subsets of the training data as targets for optimizing a Selective VQA model.
  arXiv  Detail & Related papers  (2023-06-14T21:22:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.