The Role of Natural Language Understanding in Multimodal Video-Based Dengue Diagnosis

• URL: http://arxiv.org/abs/2608.12677v1
• Date: Thu, 13 Aug 2026 00:27:22 GMT
• Title: The Role of Natural Language Understanding in Multimodal Video-Based Dengue Diagnosis
• Abstract summary: A vision-language framework is proposed to classify mosquito flight frames of uninfected and Dengue virus serotype 2 (DENV2)-infected mosquitoes.<n>The proposed method achieved 98.54% accuracy and 99.91% sensitivity at the frame level.<n>These findings suggest that vision-language models can provide a useful framework for analyzing infection-related biological behaviors from video data.
• Score: 1.5560241758540234
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Detecting infection-related behavioral changes in mosquitoes from video data is challenging because mosquitoes are small, move rapidly and irregularly, and are affected by environmental factors such as background, lighting, and shadows, which can make reliable feature extraction difficult. In this study, a YOLO- and Contrastive Language-Image Pre-training (CLIP)-based vision-language framework is proposed to classify mosquito flight frames of uninfected and Dengue virus serotype 2 (DENV2)-infected mosquitoes. First, YOLO is used to isolate mosquito regions from the background. Then, visual features extracted from video frames are aligned with biologically meaningful textual prompts in a shared embedding space. The multimodal model was fine-tuned using supervised bidirectional contrastive learning and evaluated through frame-level image-text similarity-based classification. The results show that the proposed method achieved 98.54% accuracy and 99.91% sensitivity at the frame level. After temporal aggregation of frame-level information, the model achieved complete video-level performance. The ablation results showed that fine-tuning and CLIP-based representations were essential for this domain, while the textual branch provided semantic image-text alignment rather than an accuracy advantage over the vision-only model. These findings suggest that vision-language models can provide a useful framework for analyzing infection-related biological behaviors from video data.

Related papers

• A Hybrid Framework of Vision Transformer and Gated Recurrent Unit for Detection of Mosquito Diseases [1.704431091578284]
  Identifying dengue virus-infected mosquitoes from control mosquitoes is a major challenge due to the small size and complexity of the video background.<n>In this study, a three-step framework is introduced: first, mosquitoes are identified and the background is removed using the YOLO 11M model, then visual features are extracted using the Vision Transformer (ViT), and finally the videos are classified with a convolutional GRU (ConvGRU)<n>The ConvGRU model achieved the best performance; it achieved 88.88% accuracy, 84.45% precision, 82.82% recall, and 82.81% F1 score.
  arXiv  Detail & Related papers  (2026-08-12T02:43:47Z)
• Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification [59.24009931000134]
  Multi-View Synergistic Learning (MVSL) is a unified framework that addresses adaptation paradigms, representation granularity, and disease semantic relationships.<n>MVSL decouples the adaptation of visual and textual encoders to respect their distinct representational characteristics.<n>It further introduces multi-granularity contrastive learning to explicitly model both global image semantics and localized lesion-level evidence.<n>MVSL consistently outperforms state-of-the-art methods in few-shot and zero-shot classification settings.
  arXiv  Detail & Related papers  (2026-04-27T02:41:27Z)
• A Lightweight and Explainable Vision-Language Framework for Crop Disease Visual Question Answering [0.2624902795082451]
  This work presents a lightweight vision-language framework for crop and disease identification from leaf images.<n>A two-stage training strategy is adopted to improve visual representation learning and cross-modal alignment.<n> Experimental results show high accuracy for both crop and disease identification.
  arXiv  Detail & Related papers  (2026-01-08T17:31:09Z)
• HiLa: Hierarchical Vision-Language Collaboration for Cancer Survival Prediction [55.00788339683146]
  We propose a novel Hierarchical vision-Language collaboration framework for improved survival prediction.<n> Specifically, HiLa employs pretrained feature extractors to generate hierarchical visual features from WSIs at both patch and region levels.<n>This ap-proach enables the comprehensive learning of discriminative visual features cor-responding to different survival-related attributes from prompts.
  arXiv  Detail & Related papers  (2025-07-07T02:06:25Z)
• VisText-Mosquito: A Unified Multimodal Benchmark Dataset for Visual Detection, Segmentation, and Textual Reasoning on Mosquito Breeding Sites [3.181314457277822]
  VisText-Mosquito is a multimodal dataset to support automated detection, segmentation, and reasoning for mosquito breeding site analysis.<n>The dataset includes 1,828 annotated images for object detection, 142 images for water surface segmentation, and natural language reasoning texts linked to each image.<n>For reasoning generation, we tested a range of large vision-language models (LVLMs) in both zero-shot and few-shot settings.
  arXiv  Detail & Related papers  (2025-06-17T15:24:30Z)
• Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding [45.248265283405004]
  Current models struggle to associate textual descriptions with disease regions due to inefficient attention mechanisms and a lack of fine-grained token representations.<n>We introduce Disease-Aware Prompting (DAP), which uses the explainability map of a VLM to identify the appropriate image features.<n>DAP improves visual grounding accuracy by 20.74% compared to state-of-the-art methods across three major chest X-ray datasets.
  arXiv  Detail & Related papers  (2025-05-21T05:16:45Z)
• ViKL: A Mammography Interpretation Framework via Multimodal Aggregation of Visual-knowledge-linguistic Features [54.37042005469384]
  We announce MVKL, the first multimodal mammography dataset encompassing multi-view images, detailed manifestations and reports. Based on this dataset, we focus on the challanging task of unsupervised pretraining. We propose ViKL, a framework that synergizes Visual, Knowledge, and Linguistic features.
  arXiv  Detail & Related papers  (2024-09-24T05:01:23Z)
• Domain Adaptive Synapse Detection with Weak Point Annotations [63.97144211520869]
  We present AdaSyn, a framework for domain adaptive synapse detection with weak point annotations. In the WASPSYN challenge at I SBI 2023, our method ranks the 1st place.
  arXiv  Detail & Related papers  (2023-08-31T05:05:53Z)
• Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing [53.89917396428747]
  Self-supervised learning in vision-language processing exploits semantic alignment between imaging and text modalities. We explicitly account for prior images and reports when available during both training and fine-tuning. Our approach, named BioViL-T, uses a CNN-Transformer hybrid multi-image encoder trained jointly with a text model.
  arXiv  Detail & Related papers  (2023-01-11T16:35:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.