Using Large Language Models to Assist Video Content Analysis: An Exploratory Study of Short Videos on Depression

• URL: http://arxiv.org/abs/2406.19528v2
• Date: Thu, 4 Jul 2024 13:47:41 GMT
• Title: Using Large Language Models to Assist Video Content Analysis: An Exploratory Study of Short Videos on Depression
• Authors: Jiaying Liu, Yunlong Wang, Yao Lyu, Yiheng Su, Shuo Niu, Xuhai Orson Xu, Yan Zhang,
• Abstract summary: We conduct a case study that followed a new workflow of Large Language Models (LLMs)-assisted multimodal content analysis. To test LLM's video annotation capabilities, we analyzed 203s extracted from 25 short videos about depression.
• Score: 17.357574228709346
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the growing interest in leveraging Large Language Models (LLMs) for content analysis, current studies have primarily focused on text-based content. In the present work, we explored the potential of LLMs in assisting video content analysis by conducting a case study that followed a new workflow of LLM-assisted multimodal content analysis. The workflow encompasses codebook design, prompt engineering, LLM processing, and human evaluation. We strategically crafted annotation prompts to get LLM Annotations in structured form and explanation prompts to generate LLM Explanations for a better understanding of LLM reasoning and transparency. To test LLM's video annotation capabilities, we analyzed 203 keyframes extracted from 25 YouTube short videos about depression. We compared the LLM Annotations with those of two human coders and found that LLM has higher accuracy in object and activity Annotations than emotion and genre Annotations. Moreover, we identified the potential and limitations of LLM's capabilities in annotating videos. Based on the findings, we explore opportunities and challenges for future research and improvements to the workflow. We also discuss ethical concerns surrounding future studies based on LLM-assisted video analysis.

Related papers

• Can Large Language Models Understand DL-Lite Ontologies? An Empirical Study [10.051572826948762]
  Large models (LLMs) have shown significant achievements in solving a wide range of tasks. We empirically analyze the LLMs' capability of understanding Description Logic (DL-Lite) We find that LLMs understand formal syntax and model-theoretic semantics of concepts and roles.
  arXiv  Detail & Related papers  (2024-06-25T13:16:34Z)
• LLM Inference Unveiled: Survey and Roofline Model Insights [62.92811060490876]
  Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges. Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model. This framework identifies the bottlenecks when deploying LLMs on hardware devices and provides a clear understanding of practical problems.
  arXiv  Detail & Related papers  (2024-02-26T07:33:05Z)
• Large Language Models: A Survey [69.72787936480394]
  Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data.
  arXiv  Detail & Related papers  (2024-02-09T05:37:09Z)
• Rethinking Interpretability in the Era of Large Language Models [76.1947554386879]
  Large language models (LLMs) have demonstrated remarkable capabilities across a wide array of tasks. The capability to explain in natural language allows LLMs to expand the scale and complexity of patterns that can be given to a human. These new capabilities raise new challenges, such as hallucinated explanations and immense computational costs.
  arXiv  Detail & Related papers  (2024-01-30T17:38:54Z)
• Video Understanding with Large Language Models: A Survey [97.29126722004949]
  Given the remarkable capabilities of large language models (LLMs) in language and multimodal tasks, this survey provides a detailed overview of recent advancements in video understanding. The emergent capabilities Vid-LLMs are surprisingly advanced, particularly their ability for open-ended multi-granularity reasoning. This survey presents a comprehensive study of the tasks, datasets, benchmarks, and evaluation methodologies for Vid-LLMs.
  arXiv  Detail & Related papers  (2023-12-29T01:56:17Z)
• Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity [61.54815512469125]
  This survey addresses the crucial issue of factuality in Large Language Models (LLMs) As LLMs find applications across diverse domains, the reliability and accuracy of their outputs become vital.
  arXiv  Detail & Related papers  (2023-10-11T14:18:03Z)
• Investigating Answerability of LLMs for Long-Form Question Answering [35.41413072729483]
  We focus on long-form question answering (LFQA) because it has several practical and impactful applications. We propose a question-generation method from abstractive summaries and show that generating follow-up questions from summaries of long documents can create a challenging setting.
  arXiv  Detail & Related papers  (2023-09-15T07:22:56Z)
• Sentiment Analysis in the Era of Large Language Models: A Reality Check [69.97942065617664]
  This paper investigates the capabilities of large language models (LLMs) in performing various sentiment analysis tasks. We evaluate performance across 13 tasks on 26 datasets and compare the results against small language models (SLMs) trained on domain-specific datasets.
  arXiv  Detail & Related papers  (2023-05-24T10:45:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.