ClimateVID -- Social Media Videos Analysis and Challenges Involved

• URL: http://arxiv.org/abs/2604.27968v1
• Date: Thu, 30 Apr 2026 15:00:52 GMT
• Title: ClimateVID -- Social Media Videos Analysis and Challenges Involved
• Abstract summary: We advance automated visual theme detection by assessing zero-shot and clustering capabilities on social media data.<n>We evaluate VideoChatGPT, PandaGPT, and VideoLLava using zero-shot image classification.<n>By treating clustering as a minimum cost multicut problem, we aim to uncover insightful patterns in an unsupervised manner.
• Score: 15.0442699168669
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The pervasive growth of digital content, specifically short videos on social media platforms, has significantly altered how topics are discussed and understood in public discourse. In this work, we advance automated visual theme detection by assessing zero-shot and clustering capabilities on social media data. (1) We evaluated the capabilities of notable VLMs such as VideoChatGPT, PandaGPT, and VideoLLava using zero-shot image classification and compared their performance to the baseline provided by frame-wise CLIP image classification. (2) By treating clustering as a minimum cost multicut problem, we aim to uncover insightful patterns in an unsupervised manner. For both analysis strategies, we provide extensive evaluations and practical guidance to practitioners. While VLMs are currently not able to detect climate change specific classes, the clustering results are distinct visual frames. %Given that VLMs are not currently capable to grasp the climate change discourse, we focus the clustering evaluation of image embedding models. We find that both ConvNeXt V2 and DINOv2 produce meaningful clusters, with DINOv2 focusing more on style differences and abstract categories, while ConvNeXt V2 clusters differ in more fine-grained ways. Code available at https://github.com/KathPra/ClimateVID.git.

Related papers

• From Codebooks to VLMs: Evaluating Automated Visual Discourse Analysis for Climate Change on Social Media [22.261744577934554]
  We aim to facilitate such research by analysing how computer vision methods can be used for social media discourse analysis.<n>This analysis includes application-based taxonomy design, model selection, prompt engineering, and validation.<n>We benchmark six promptable vision-language models and 15 zero-shot CLIP-like models on two datasets from X (formerly Twitter)
  arXiv  Detail & Related papers  (2026-04-23T15:44:14Z)
• VLIC: Vision-Language Models As Perceptual Judges for Human-Aligned Image Compression [83.36460501519203]
  Vision-Language Models for Image Compression (VLIC) is a diffusion-based image compression system designed to be post-trained with binary VLM judgments.<n>We show that calibrating this system on VLM judgments produces competitive or state-of-the-art performance on human-aligned visual compression depending on the dataset.
  arXiv  Detail & Related papers  (2025-12-17T18:52:55Z)
• Image Recognition with Vision and Language Embeddings of VLMs [14.022566577479322]
  Vision-language models (VLMs) have enabled strong zero-shot classification through image-text alignment.<n>We conduct a comprehensive evaluation of both language-guided and vision-only image classification with a diverse set of dual-encoder VLMs.<n>Key factors affecting accuracy are analysed, including prompt design, class diversity, the number of neighbours in k-NN, and reference set size.
  arXiv  Detail & Related papers  (2025-09-11T09:54:25Z)
• I Spy With My Little Eye: A Minimum Cost Multicut Investigation of Dataset Frames [12.177674038614658]
  Visual framing analysis is a key method in social sciences for determining common themes and concepts in a discourse.<n>In this work, we phrase the clustering task as a Minimum Cost Multicut Problem [MP]<n> Solutions to the MP have been shown to provide clusterings that maximize the posterior probability, solely from provided local, pairwise probabilities of two images belonging to the same cluster.<n>Our insights into embedding space differences in combination with the optimal clustering - by definition - advances automated visual frame detection.
  arXiv  Detail & Related papers  (2024-12-02T09:09:47Z)
• VHELM: A Holistic Evaluation of Vision Language Models [75.88987277686914]
  We present the Holistic Evaluation of Vision Language Models (VHELM) VHELM aggregates various datasets to cover one or more of the 9 aspects: visual perception, knowledge, reasoning, bias, fairness, multilinguality, robustness, toxicity, and safety. Our framework is designed to be lightweight and automatic so that evaluation runs are cheap and fast.
  arXiv  Detail & Related papers  (2024-10-09T17:46:34Z)
• Why are Visually-Grounded Language Models Bad at Image Classification? [39.76294811955341]
  We revisit the image classification task using visually-grounded language models (VLMs) such as GPT-4V and LLaVA. We find that existing proprietary and public VLMs significantly underperform CLIP on standard image classification benchmarks like ImageNet. Our analysis reveals that the primary cause is data-related: critical information for image classification is encoded in the VLM's latent space but can only be effectively decoded with enough training data.
  arXiv  Detail & Related papers  (2024-05-28T17:57:06Z)
• Image as Set of Points [60.30495338399321]
  Context clusters (CoCs) view an image as a set of unorganized points and extract features via simplified clustering algorithm. Our CoCs are convolution- and attention-free, and only rely on clustering algorithm for spatial interaction.
  arXiv  Detail & Related papers  (2023-03-02T18:56:39Z)
• GOCA: Guided Online Cluster Assignment for Self-Supervised Video Representation Learning [49.69279760597111]
  Clustering is a ubiquitous tool in unsupervised learning. Most of the existing self-supervised representation learning methods typically cluster samples based on visually dominant features. We propose a principled way to combine two views. Specifically, we propose a novel clustering strategy where we use the initial cluster assignment of each view as prior to guide the final cluster assignment of the other view.
  arXiv  Detail & Related papers  (2022-07-20T19:26:55Z)
• Unsupervised Learning of Video Representations via Dense Trajectory Clustering [86.45054867170795]
  This paper addresses the task of unsupervised learning of representations for action recognition in videos. We first propose to adapt two top performing objectives in this class - instance recognition and local aggregation. We observe promising performance, but qualitative analysis shows that the learned representations fail to capture motion patterns.
  arXiv  Detail & Related papers  (2020-06-28T22:23:03Z)
• Unsupervised Person Re-identification via Softened Similarity Learning [122.70472387837542]
  Person re-identification (re-ID) is an important topic in computer vision. This paper studies the unsupervised setting of re-ID, which does not require any labeled information. Experiments on two image-based and video-based datasets demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-04-07T17:16:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.