LLM4VG: Large Language Models Evaluation for Video Grounding
- URL: http://arxiv.org/abs/2312.14206v2
- Date: Thu, 28 Dec 2023 13:02:31 GMT
- Title: LLM4VG: Large Language Models Evaluation for Video Grounding
- Authors: Wei Feng, Xin Wang, Hong Chen, Zeyang Zhang, Zihan Song, Yuwei Zhou,
Wenwu Zhu
- Abstract summary: This paper systematically evaluates the performance of different LLMs on video grounding tasks.
We propose prompt methods to integrate the instruction of VG and description from different kinds of generators.
Our experimental evaluations lead to two conclusions: (i) the existing VidLLMs are still far away from achieving satisfactory video grounding performance, and more time-related video tasks should be included to further fine-tune these models.
- Score: 45.94959878409729
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Recently, researchers have attempted to investigate the capability of LLMs in
handling videos and proposed several video LLM models. However, the ability of
LLMs to handle video grounding (VG), which is an important time-related video
task requiring the model to precisely locate the start and end timestamps of
temporal moments in videos that match the given textual queries, still remains
unclear and unexplored in literature. To fill the gap, in this paper, we
propose the LLM4VG benchmark, which systematically evaluates the performance of
different LLMs on video grounding tasks. Based on our proposed LLM4VG, we
design extensive experiments to examine two groups of video LLM models on video
grounding: (i) the video LLMs trained on the text-video pairs (denoted as
VidLLM), and (ii) the LLMs combined with pretrained visual description models
such as the video/image captioning model. We propose prompt methods to
integrate the instruction of VG and description from different kinds of
generators, including caption-based generators for direct visual description
and VQA-based generators for information enhancement. We also provide
comprehensive comparisons of various VidLLMs and explore the influence of
different choices of visual models, LLMs, prompt designs, etc, as well. Our
experimental evaluations lead to two conclusions: (i) the existing VidLLMs are
still far away from achieving satisfactory video grounding performance, and
more time-related video tasks should be included to further fine-tune these
models, and (ii) the combination of LLMs and visual models shows preliminary
abilities for video grounding with considerable potential for improvement by
resorting to more reliable models and further guidance of prompt instructions.
Related papers
- From Image to Video, what do we need in multimodal LLMs? [19.85928004619801]
Multimodal Large Language Models (MLLMs) have demonstrated profound capabilities in understanding multimodal information.
We propose RED-VILLM, a Resource-Efficient Development pipeline for Video LLMs from Image LLMs.
Our approach highlights the potential for a more cost-effective and scalable advancement in multimodal models.
arXiv Detail & Related papers (2024-04-18T02:43:37Z) - ST-LLM: Large Language Models Are Effective Temporal Learners [58.79456373423189]
Large Language Models (LLMs) have showcased impressive capabilities in text comprehension and generation.
How to effectively encode and understand videos in video-based dialogue systems remains to be solved.
We propose ST-LLM, an effective video-LLM baseline with spatial-temporal sequence modeling inside LLM.
arXiv Detail & Related papers (2024-03-30T10:11:26Z) - TempCompass: Do Video LLMs Really Understand Videos? [36.28973015469766]
Existing benchmarks fail to provide a comprehensive feedback on the temporal perception ability of Video LLMs.
We propose the textbfTemp benchmark, which introduces a diversity of high-quality temporal aspects and task formats.
arXiv Detail & Related papers (2024-03-01T12:02:19Z) - OPDAI at SemEval-2024 Task 6: Small LLMs can Accelerate Hallucination
Detection with Weakly Supervised Data [1.3981625092173873]
This paper describes a unified system for hallucination detection of LLMs.
It wins the second prize in the model-agnostic track of the SemEval-2024 Task 6.
arXiv Detail & Related papers (2024-02-20T11:01:39Z) - Video Understanding with Large Language Models: A Survey [97.29126722004949]
Given the remarkable capabilities of large language models (LLMs) in language and multimodal tasks, this survey provides a detailed overview of recent advancements in video understanding.
The emergent capabilities Vid-LLMs are surprisingly advanced, particularly their ability for open-ended multi-granularity reasoning.
This survey presents a comprehensive study of the tasks, datasets, benchmarks, and evaluation methodologies for Vid-LLMs.
arXiv Detail & Related papers (2023-12-29T01:56:17Z) - VideoLLM: Modeling Video Sequence with Large Language Models [70.32832021713864]
Existing video understanding models are often task-specific and lack a comprehensive capability of handling diverse tasks.
We propose a novel framework called VideoLLM that leverages the sequence reasoning capabilities of pre-trained LLMs.
VideoLLM incorporates a carefully designed Modality and Semantic Translator, which convert inputs from various modalities into a unified token sequence.
arXiv Detail & Related papers (2023-05-22T17:51:22Z) - mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality [95.76661165594884]
mPLUG-Owl is a training paradigm that equips large language models (LLMs) with multi-modal abilities.
The training paradigm involves a two-stage method for aligning image and text, which learns visual knowledge with the assistance of LLM.
Experimental results show that our model outperforms existing multi-modal models.
arXiv Detail & Related papers (2023-04-27T13:27:01Z) - VALUE: A Multi-Task Benchmark for Video-and-Language Understanding
Evaluation [124.02278735049235]
VALUE benchmark aims to cover a broad range of video genres, video lengths, data volumes, and task difficulty levels.
We evaluate various baseline methods with and without large-scale VidL pre-training.
The significant gap between our best model and human performance calls for future study for advanced VidL models.
arXiv Detail & Related papers (2021-06-08T18:34:21Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.