A Simple LLM Framework for Long-Range Video Question-Answering
- URL: http://arxiv.org/abs/2312.17235v2
- Date: Mon, 26 Feb 2024 17:29:30 GMT
- Title: A Simple LLM Framework for Long-Range Video Question-Answering
- Authors: Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu,
Mohit Bansal, Gedas Bertasius
- Abstract summary: We present LLoVi, a language-based framework for long-range video question-answering (LVQA)
Our approach uses a frame/clip-level visual captioner coupled with a Large Language Model (GPT-3.5, GPT-4)
Our method achieves 50.3% accuracy, outperforming the previous best-performing approach by 18.1% (absolute gain)
- Score: 66.68887077133355
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We present LLoVi, a language-based framework for long-range video
question-answering (LVQA). Unlike prior long-range video understanding methods,
which are often costly and require specialized long-range video modeling design
(e.g., memory queues, state-space layers, etc.), our approach uses a
frame/clip-level visual captioner (e.g., BLIP2, LaViLa, LLaVA) coupled with a
Large Language Model (GPT-3.5, GPT-4) leading to a simple yet surprisingly
effective LVQA framework. Specifically, we decompose short and long-range
modeling aspects of LVQA into two stages. First, we use a short-term visual
captioner to generate textual descriptions of short video clips (0.5-8s in
length) densely sampled from a long input video. Afterward, an LLM aggregates
the densely extracted short-term captions to perform long-range temporal
reasoning needed to understand the whole video and answer a question. To
analyze what makes our simple framework so effective, we thoroughly evaluate
various components of our system. Our empirical analysis reveals that the
choice of the visual captioner and LLM is critical for good LVQA performance.
Furthermore, we show that a specialized prompt that asks the LLM first to
summarize the noisy short-term visual captions and then answer a given input
question leads to a significant LVQA performance boost. On EgoSchema, which is
best known as a very long-form video question-answering benchmark, our method
achieves 50.3% accuracy, outperforming the previous best-performing approach by
18.1% (absolute gain). In addition, our approach outperforms the previous
state-of-the-art by 4.1% and 3.1% on NeXT-QA and IntentQA. We also extend LLoVi
to grounded LVQA and show that it outperforms all prior methods on the NeXT-GQA
dataset. We will release our code at https://github.com/CeeZh/LLoVi.
Related papers
- Goldfish: Vision-Language Understanding of Arbitrarily Long Videos [51.547065479762715]
We present a methodology tailored for comprehending videos of arbitrary lengths.
We also introduce the TVQA-long benchmark, designed to evaluate models' capabilities in understanding long videos with questions in both vision and text content.
Our results indicate that our models have significant improvements in both long and short-video understanding.
arXiv Detail & Related papers (2024-07-17T15:59:32Z) - Hallucination Mitigation Prompts Long-term Video Understanding [36.26790392889717]
This paper constructs a comprehensive hallucination mitigation pipeline based on existing MLLMs.
We use the CLIP Score to guide the frame sampling process with questions, selecting key frames relevant to the question.
During the answer generation stage, we utilize chain-of-thought and in-context learning techniques to explicitly control the generation of answers.
arXiv Detail & Related papers (2024-06-17T08:44:03Z) - Too Many Frames, not all Useful:Efficient Strategies for Long-Form Video QA [40.54207548074378]
Long-form videos that span across wide temporal intervals are highly information redundant.
When performing long-form video question answering (LVQA),all information necessary to generate a correct response can often be contained within a small subset of frames.
Recent literature explore the use of large language models (LLMs) in LVQA benchmarks, achieving exceptional performance.
arXiv Detail & Related papers (2024-06-13T17:59:16Z) - VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos [67.78336281317347]
VideoTree is a queryadaptive and hierarchical framework for long-video understanding with Large Language Models.
VideoTree adaptively selects frames for captioning by iteratively clustering frames based on their visual features.
It organizes visual clusters into a query-adaptive and hierarchical tree structure.
arXiv Detail & Related papers (2024-05-29T15:49:09Z) - Koala: Key frame-conditioned long video-LLM [70.52369588364992]
We propose a lightweight and self-supervised long video-LLM (Koala) to adapt pretrained vLLMs for generalizing to longer videos.
Our approach outperforms state-of-the-art large models by 3 - 6% in absolute accuracy across all tasks.
Surprisingly, we also empirically show that our approach not only helps a pretrained vLLM to understand long videos but also improves its accuracy on short-term action recognition.
arXiv Detail & Related papers (2024-04-05T18:33:04Z) - VaQuitA: Enhancing Alignment in LLM-Assisted Video Understanding [63.075626670943116]
We introduce a cutting-edge framework, VaQuitA, designed to refine the synergy between video and textual information.
At the data level, instead of sampling frames uniformly, we implement a sampling method guided by CLIP-score rankings.
At the feature level, we integrate a trainable Video Perceiver alongside a Visual-Query Transformer.
arXiv Detail & Related papers (2023-12-04T19:48:02Z) - Natural Language Video Localization: A Revisit in Span-based Question
Answering Framework [56.649826885121264]
Natural Language Video Localization (NLVL) aims to locate a target moment from an untrimmed video that semantically corresponds to a text query.
Existing approaches mainly solve the NLVL problem from the perspective of computer vision.
We address the NLVL from a new perspective, i.e., span-based question answering (QA), by treating the input video as a text passage.
arXiv Detail & Related papers (2021-02-26T15:57:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.