DementiaCare-Bench: A Modality-Validated Video Benchmark

• URL: http://arxiv.org/abs/2609.12929v1
• Date: Fri, 11 Sep 2026 14:52:49 GMT
• Title: DementiaCare-Bench: A Modality-Validated Video Benchmark
• Abstract summary: Dementia affects an estimated 57 million people worldwide.<n>For most families the hardest part of care is not memory loss but the behavioral and psychological symptoms of dementia.<n>We present DementiaCare-Bench: 56 professionally produced caregiver training videos segmented into 94 clips across nine BPSD categories.
• Score: 83.92703885588868
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dementia affects an estimated 57 million people worldwide, and for most families the hardest part of care is not memory loss but the behavioral and psychological symptoms of dementia (BPSD): agitation, wandering, resistance to care, sundowning. Understanding these symptoms requires more than recognizing the behavior itself; it also requires knowing what happened beforehand. The same behavior may call for a different response depending on its trigger. Video-language models (VLMs) could potentially support caregivers, yet no existing benchmark evaluates this capability. To fill this gap, we present DementiaCare-Bench: 56 professionally produced caregiver training videos segmented into 94 clips across nine BPSD categories, with 2023 questions generated by a multi-agent pipeline that grounds every clinical claim in a verbatim transcript span. Each question is then probed under four visual conditions and labelled by the least it requires, so its visual demand is measured rather than assumed. Measurement contradicts intent: we wrote 77.7% of the questions to require ordered frames, and 34.8% do. Across 12 current VLMs the pattern is uniform. The best reach 85% overall, but that average is carried by questions a language model can answer from clinical knowledge alone; accuracy falls by 17 points on average on questions that require the ordered clip, and a leading open model scores at chance on judging whether a caregiver's response was appropriate. A lightweight LoRA fine-tune, DemCare-VLM, moves video dependence from -3.3 to +4.5 points, so what the benchmark exposes can be repaired and not only measured.

Related papers

• SDoH-Aware Narrative Anchoring Bias in Medical LLMs for Trustworthy Clinical Decision Support [0.0]
  This paper evaluates the risk as SDoH aware narrative anchoring bias.<n>We use NarrativeShield SDoH MedQA, a counterfactual medical question answering dataset.<n>The final experiment uses 300 clinical cases and produces 8,100 model responses.
  arXiv  Detail & Related papers  (2026-08-24T04:58:32Z)
• Trivial Prompt Reframing Bypasses Safety Guardrails in Googleś MedGemma-4B [0.0]
  Open-weight medical language models are increasingly used as the base of patient-facing and clinician-support applications.<n>We quantify that gap for MedGemma-4B-it under attacks that require no technical sophistication.
  arXiv  Detail & Related papers  (2026-07-09T18:31:43Z)
• Measuring Epistemic Resilience of LLMs Under Misleading Medical Context [47.20527783144983]
  Large language models (LLMs) now reach expert-level scores on medical licensing exams.<n>We show this assumption is fragile: when misleading context is injected into questions that LLMs originally answer correctly, they abandon the correct answer.<n>We call the ability to maintain correct judgment under adversarial context epistemic resilience, and introduce MedMisBench to measure it.
  arXiv  Detail & Related papers  (2026-06-10T16:27:26Z)
• ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV [0.0]
  Reasoning benchmarks measure clinical performance on clean inputs.<n>We evaluate the step before reasoning: retrieval over real EHR notes, where negation, temporality, and family-versus-patient attribution can flip a correct answer to a wrong one.<n>EpiKG carries assertion label and a temporality tag with every fact in a patient knowledge graph, then routes retrieval by question intent.
  arXiv  Detail & Related papers  (2026-05-11T18:47:52Z)
• ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models [60.14219417402433]
  Existing memory benchmarks for LLM agents evaluate explicit recall of facts, yet overlook implicit memory where experience becomes automated behavior without conscious retrieval.<n>We introduce ImplicitMemBench, the first systematic benchmark evaluating implicit memory through three constructs drawn from standard cognitive-science accounts of non-declarative memory.<n>Our 300-item suite employs a unified Learning/Priming-Interfere-Test protocol with first-attempt scoring.
  arXiv  Detail & Related papers  (2026-04-09T10:26:32Z)
• PSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language Models [2.064612766965483]
  We introduce Paraphrase Sensitivity Failure (PSF)-Med, a benchmark of 19,748 chest Xray questions paired with about 92,000 paraphrases.<n>We measure yes/no flips for the same image and find flip rates from 8% to 58%.<n>We identify a sparse feature at layer 17 that correlates with prompt framing and predicts decision margin shifts.
  arXiv  Detail & Related papers  (2026-02-24T23:03:50Z)
• MedOmni-45°: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine [69.08855631283829]
  We introduce Med Omni-45 Degrees, a benchmark designed to quantify safety-performance trade-offs under manipulative hint conditions.<n>It contains 1,804 reasoning-focused medical questions across six specialties and three task types, including 500 from MedMCQA.<n>Results show a consistent safety-performance trade-off, with no model surpassing the diagonal.
  arXiv  Detail & Related papers  (2025-08-22T08:38:16Z)
• Medical Hallucinations in Foundation Models and Their Impact on Healthcare [71.15392179084428]
  Hallucinations in foundation models arise from autoregressive training objectives.<n>Top-performing models exceeded 97% accuracy when augmented with chain-of-thought prompting.
  arXiv  Detail & Related papers  (2025-02-26T02:30:44Z)
• MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records [60.35217378132709]
  Large language models (LLMs) can follow natural language instructions with human-level fluency. evaluating LLMs on realistic text generation tasks for healthcare remains challenging. We introduce MedAlign, a benchmark dataset of 983 natural language instructions for EHR data.
  arXiv  Detail & Related papers  (2023-08-27T12:24:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.