VIS-DICT: A Visual Dictionary for Missing Modality Imputation in Social Network Depression Detection

• URL: http://arxiv.org/abs/2609.05537v1
• Date: Wed, 02 Sep 2026 08:13:30 GMT
• Title: VIS-DICT: A Visual Dictionary for Missing Modality Imputation in Social Network Depression Detection
• Abstract summary: Tracking social media posts can help spot early signs of depression.<n>Many social media posts do not have images, which makes it hard to use multimodal models.<n>We introduce Vis-Dict, a method that builds missing visual features by linking words to average image vectors from complete training posts.
• Score: 0.17205106391379021
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tracking social media posts can help spot early signs of depression. Recent studies show that combining text and images works better for detecting depression than using text alone. However, many social media posts do not have images, which makes it hard to use multimodal models. Most existing methods fill in missing images using retrieval or generative models that need extra training. In this paper, we introduce Vis-Dict, a dictionary-based method that builds missing visual features by linking words to average image vectors from complete training posts. These estimated visual features are then combined with text to track changes in user behavior over time. We tested Vis-Dict on a social media dataset using user timelines of up to 512 posts and compared it with other missing-data methods. The results show that Vis-Dict performs on par with generative networks, reaching an F1-score of 0.9454 and an ROC-AUC of 0.9890. Most importantly, Vis-Dict achieves this strong performance with zero trainable parameters for image generation. These findings show that directly connecting words to visual features is an effective and practical way to handle missing images in depression detection systems.

Related papers

• The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric [78.74729751232852]
  We introduce a large-scale dataset of human similarity judgments over image triplets.<n>We fine-tune a vision-language model to produce our Text-Prompted Image Perceptual Similarity metric.<n>TPIPS aligns more closely with human perception and generalizes reliably beyond the training distribution.
  arXiv  Detail & Related papers  (2026-07-20T17:59:51Z)
• Towards Better Multi-modal Keyphrase Generation via Visual Entity Enhancement and Multi-granularity Image Noise Filtering [79.44443231700201]
  Multi-modal keyphrase generation aims to produce a set of keyphrases that represent the core points of the input text-image pair. The input text and image are often not perfectly matched, and thus the image may introduce noise into the model. We propose a novel multi-modal keyphrase generation model, which not only enriches the model input with external knowledge, but also effectively filters image noise.
  arXiv  Detail & Related papers  (2023-09-09T09:41:36Z)
• Improving Multimodal Datasets with Image Captioning [65.74736570293622]
  We study how generated captions can increase the utility of web-scraped datapoints with nondescript text. Our experiments with using generated captions at DataComp's large scale (1.28B image-text pairs) offer insights into the limitations of synthetic text.
  arXiv  Detail & Related papers  (2023-07-19T17:47:12Z)
• Revisiting the Role of Language Priors in Vision-Language Models [90.0317841097143]
  Vision-language models (VLMs) are applied to a variety of visual understanding tasks in a zero-shot fashion, without any fine-tuning. We study $textitgenerative VLMs$ that are trained for next-word generation given an image. We explore their zero-shot performance on the illustrative task of image-text retrieval across 8 popular vision-language benchmarks.
  arXiv  Detail & Related papers  (2023-06-02T19:19:43Z)
• Scene Text Recognition with Image-Text Matching-guided Dictionary [17.073688809336456]
  We propose a new dictionary language model leveraging the Scene Image-Text Matching(SITM) network. Inspired by ITC, the SITM network combines the visual features and the text features of all candidates to identify the candidate with the minimum distance in the feature space. Our lexicon method achieves better results(93.8% accuracy) than the ordinary method results(92.1% accuracy) on six mainstream benchmarks.
  arXiv  Detail & Related papers  (2023-05-08T07:47:49Z)
• Borrowing Human Senses: Comment-Aware Self-Training for Social Media Multimodal Classification [5.960550152906609]
  We capture hinting features from user comments, which are retrieved via jointly leveraging visual and lingual similarity. The classification tasks are explored via self-training in a teacher-student framework, motivated by the usually limited labeled data scales. The results show that our method further advances the performance of previous state-of-the-art models.
  arXiv  Detail & Related papers  (2023-03-27T08:59:55Z)
• It's Just a Matter of Time: Detecting Depression with Time-Enriched Multimodal Transformers [24.776445591293186]
  We propose a flexible time-enriched multimodal transformer architecture for detecting depression from social media posts. Our model operates directly at the user-level, and we enrich it with the relative time between posts by using time2vec positional embeddings. We show that our method, using EmoBERTa and CLIP embeddings, surpasses other methods on two multimodal datasets.
  arXiv  Detail & Related papers  (2023-01-13T09:40:19Z)
• NewsStories: Illustrating articles with visual summaries [49.924916589209374]
  We introduce a large-scale multimodal dataset containing over 31M articles, 22M images and 1M videos. We show that state-of-the-art image-text alignment methods are not robust to longer narratives with multiple images. We introduce an intuitive baseline that outperforms these methods on zero-shot image-set retrieval by 10% on the GoodNews dataset.
  arXiv  Detail & Related papers  (2022-07-26T17:34:11Z)
• Learning to Generate Scene Graph from Natural Language Supervision [52.18175340725455]
  We propose one of the first methods that learn from image-sentence pairs to extract a graphical representation of localized objects and their relationships within an image, known as scene graph. We leverage an off-the-shelf object detector to identify and localize object instances, match labels of detected regions to concepts parsed from captions, and thus create "pseudo" labels for learning scene graph.
  arXiv  Detail & Related papers  (2021-09-06T03:38:52Z)
• Deep Multimodal Image-Text Embeddings for Automatic Cross-Media Retrieval [0.0]
  We introduce an end-to-end deep multimodal convolutional-recurrent network for learning both vision and language representations simultaneously. The model learns which pairs are a match (positive) and which ones are a mismatch (negative) using a hinge-based triplet ranking.
  arXiv  Detail & Related papers  (2020-02-23T23:58:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.