Content-Rich AIGC Video Quality Assessment via Intricate Text Alignment and Motion-Aware Consistency

• URL: http://arxiv.org/abs/2502.04076v1
• Date: Thu, 06 Feb 2025 13:41:24 GMT
• Title: Content-Rich AIGC Video Quality Assessment via Intricate Text Alignment and Motion-Aware Consistency
• Authors: Shangkun Sun, Xiaoyu Liang, Bowen Qu, Wei Gao,
• Abstract summary: We propose textbfCRAVE (underlineContent-underlineRich underlineAIGC underlineVideo underlineEvaluator) for the evaluation of Sora-era AIGC videos.
• Score: 4.922783970210658
• License:
• Abstract: The advent of next-generation video generation models like \textit{Sora} poses challenges for AI-generated content (AIGC) video quality assessment (VQA). These models substantially mitigate flickering artifacts prevalent in prior models, enable longer and complex text prompts and generate longer videos with intricate, diverse motion patterns. Conventional VQA methods designed for simple text and basic motion patterns struggle to evaluate these content-rich videos. To this end, we propose \textbf{CRAVE} (\underline{C}ontent-\underline{R}ich \underline{A}IGC \underline{V}ideo \underline{E}valuator), specifically for the evaluation of Sora-era AIGC videos. CRAVE proposes the multi-granularity text-temporal fusion that aligns long-form complex textual semantics with video dynamics. Additionally, CRAVE leverages the hybrid motion-fidelity modeling to assess temporal artifacts. Furthermore, given the straightforward prompts and content in current AIGC VQA datasets, we introduce \textbf{CRAVE-DB}, a benchmark featuring content-rich videos from next-generation models paired with elaborate prompts. Extensive experiments have shown that the proposed CRAVE achieves excellent results on multiple AIGC VQA benchmarks, demonstrating a high degree of alignment with human perception. All data and code will be publicly available at https://github.com/littlespray/CRAVE.

Related papers

• Generative Ghost: Investigating Ranking Bias Hidden in AI-Generated Videos [106.5804660736763]
  Video information retrieval remains a fundamental approach for accessing video content. We build on the observation that retrieval models often favor AI-generated content in ad-hoc and image retrieval tasks. We investigate whether similar biases emerge in the context of challenging video retrieval.
  arXiv  Detail & Related papers  (2025-02-11T07:43:47Z)
• Multilevel Semantic-Aware Model for AI-Generated Video Quality Assessment [15.529169236891532]
  We introduce MSA-VQA, a Multilevel Semantic-Aware Model for AI-Generated Video Quality Assessment. Our hierarchical framework analyzes video content at three levels: frame, segment, and video. We propose a Prompt Semantic Supervision Module using text encoder of CLIP to ensure semantic consistency between videos and conditional prompts.
  arXiv  Detail & Related papers  (2025-01-06T01:18:11Z)
• Track the Answer: Extending TextVQA from Image to Video with Spatio-Temporal Clues [8.797350517975477]
  Video text-based visual question answering (Video TextVQA) is a practical task that aims to answer questions by jointly textual reasoning and visual information in a given video. We propose the TEA (stands for textbfTrack thbfE bftextAlanguageer'') method that better extends the generative TextVQA framework from image to video.
  arXiv  Detail & Related papers  (2024-12-17T03:06:12Z)
• Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification [5.468979600421325]
  We introduce NeuS-V, a novel synthetic video evaluation metric. NeuS-V rigorously assesses text-to-video alignment using neuro-symbolic formal verification techniques. We find that NeuS-V demonstrates a higher correlation by over 5x with human evaluations when compared to existing metrics.
  arXiv  Detail & Related papers  (2024-11-22T23:59:12Z)
• VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection [61.54044967253421]
  We introduce VideoEspresso, a novel dataset that features VideoQA pairs preserving essential spatial details and temporal coherence. Our construction pipeline employs a semantic-aware method to reduce redundancy, followed by generating QA pairs using GPT-4o. We propose a Hybrid LVLMs Collaboration framework, featuring a Frame Selector and a two-stage instruction fine-tuned reasoning LVLM.
  arXiv  Detail & Related papers  (2024-11-22T08:33:36Z)
• VERIFIED: A Video Corpus Moment Retrieval Benchmark for Fine-Grained Video Understanding [44.382937324454254]
  Existing Video Corpus Moment Retrieval (VCMR) is limited to coarse-grained understanding. We propose a more challenging fine-grained VCMR benchmark requiring methods to localize the best-matched moment from the corpus. With VERIFIED, we construct a more challenging fine-grained VCMR benchmark containing Charades-FIG, DiDeMo-FIG, and ActivityNet-FIG.
  arXiv  Detail & Related papers  (2024-10-11T07:42:36Z)
• xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations [120.52120919834988]
  xGen-SynVideo-1 is a text-to-video (T2V) generation model capable of producing realistic scenes from textual descriptions. VidVAE compresses video data both spatially and temporally, significantly reducing the length of visual tokens. DiT model incorporates spatial and temporal self-attention layers, enabling robust generalization across different timeframes and aspect ratios.
  arXiv  Detail & Related papers  (2024-08-22T17:55:22Z)
• RTQ: Rethinking Video-language Understanding Based on Image-text Model [55.278942477715084]
  Video-language understanding presents unique challenges due to the inclusion of highly complex semantic details. We propose a novel framework called RTQ, which addresses these challenges simultaneously. Our model demonstrates outstanding performance even in the absence of video-language pre-training.
  arXiv  Detail & Related papers  (2023-12-01T04:51:01Z)
• Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation [55.36617538438858]
  We propose a novel approach that strengthens the interaction between spatial and temporal perceptions. We curate a large-scale and open-source video dataset called HD-VG-130M.
  arXiv  Detail & Related papers  (2023-05-18T11:06:15Z)
• Hierarchical Conditional Relation Networks for Multimodal Video Question Answering [67.85579756590478]
  Video QA adds at least two more layers of complexity - selecting relevant content for each channel in the context of a linguistic query. Conditional Relation Network (CRN) takes as input a set of tensorial objects translating into a new set of objects that encode relations of the inputs. CRN is then applied for Video QA in two forms, short-form where answers are reasoned solely from the visual content, and long-form where associated information, such as subtitles, is presented.
  arXiv  Detail & Related papers  (2020-10-18T02:31:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.