DementiaCare-Bench: A Modality-Validated Video Benchmark
Abstract Overview
DementiaCare-Bench is a video question-answering benchmark for dementia caregiving, built from 56 publicly available professional training videos segmented into 94 clips across nine behavioral and psychological symptom categories. The benchmark contains 2,023 multiple-choice questions spanning visual perception, temporal and causal reasoning, clinical reasoning, and comparative analysis, with clinical claims grounded in transcript evidence. A key methodological feature is modality validation: every question is tested under no-frame, single-frame, shuffled-frame, and ordered-frame conditions so that its actual dependence on video is measured rather than assumed. This validation reveals a substantial gap between intended and verified temporal demands, providing a rigorous testbed for evaluating whether video-language models genuinely utilize visual evidence in caregiving contexts.
Novelty
The paper introduces a dementia-care video benchmark featuring systematic modality validation, certifying each question by the minimal visual information required to solve it rather than relying on design intent. It also establishes a paired-clip structure to assess caregiver response appropriateness and introduces DemCare-VLM, a lightweight fine-tuning approach trained with reject-on-mismatch negative samples to mitigate language-prior shortcuts.
Results
Evaluation of 12 video-language models showed that high aggregate scores were largely sustained by text-alone priors, with average accuracy dropping by approximately 17 points on questions verified to require ordered frames. Modality probing revealed that while 77.7% of questions were authored to require ordered video, only 34.8% actually did. Applying DemCare-VLM fine-tuning shifted video dependence on a held-out split from -3.3 to +4.5 points, demonstrating increased conditioning on visual clip evidence.
Key Points
- DementiaCare-Bench consists of 94 clips derived from 56 professional caregiver-training videos, spanning nine BPSD categories and 2,023 grounded multiple-choice questions.
- Per-question modality validation across four visual conditions demonstrated that more than half of nominally temporal questions could be answered with reduced or no visual input.
- Current state-of-the-art VLMs heavily exploit language priors on caregiving tasks, but LoRA fine-tuning with mismatch negatives (DemCare-VLM) effectively improves model reliance on visual context.