Transforming faces into video stories -- VideoFace2.0

• URL: http://arxiv.org/abs/2505.02060v2
• Date: Thu, 08 May 2025 07:41:06 GMT
• Title: Transforming faces into video stories -- VideoFace2.0
• Authors: Branko Brkljač, Vladimir Kalušev, Branislav Popović, Milan Sečujski,
• Abstract summary: VideoFace2.0 is the name of the developed system for spatial and temporal localization of each unique face in the input video.<n>The proposed algorithm brings relative gain in the reduction of number of false identities in the range of 73%-93%.<n>The system is envisioned as a compact and modular extensions of the existing video production equipment.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Face detection and face recognition have been in the focus of vision community since the very beginnings. Inspired by the success of the original Videoface digitizer, a pioneering device that allowed users to capture video signals from any source, we have designed an advanced video analytics tool to efficiently create structured video stories, i.e. identity-based information catalogs. VideoFace2.0 is the name of the developed system for spatial and temporal localization of each unique face in the input video, i.e. face re-identification (ReID), which also allows their cataloging, characterization and creation of structured video outputs for later downstream tasks. Developed near real-time solution is primarily designed to be utilized in application scenarios involving TV production, media analysis, and as an efficient tool for creating large video datasets necessary for training machine learning (ML) models in challenging vision tasks such as lip reading and multimodal speech recognition. Conducted experiments confirm applicability of the proposed face ReID algorithm that is combining the concepts of face detection, face recognition and passive tracking-by-detection in order to achieve robust and efficient face ReID. The system is envisioned as a compact and modular extensions of the existing video production equipment. Presented results are based on test implementation that achieves between 18-25 fps on consumer type notebook. Ablation experiments also confirmed that the proposed algorithm brings relative gain in the reduction of number of false identities in the range of 73%-93%. We hope that the presented work and shared code implementation will stimulate further interest in development of similar, application specific video analysis tools, and lower the entry barrier for production of high-quality multi-modal datasets in the future.

Related papers

• Leveraging Pre-Trained Visual Models for AI-Generated Video Detection [54.88903878778194]
  The field of video generation has advanced beyond DeepFakes, creating an urgent need for methods capable of detecting AI-generated videos with generic content.<n>We propose a novel approach that leverages pre-trained visual models to distinguish between real and generated videos.<n>Our method achieves high detection accuracy, above 90% on average, underscoring its effectiveness.
  arXiv  Detail & Related papers  (2025-07-17T15:36:39Z)
• Spotting tell-tale visual artifacts in face swapping videos: strengths and pitfalls of CNN detectors [2.89209645531276]
  Face swapping manipulations in video streams represents an increasing threat in remote video communications.<n>Recent literature proposes to characterize and exploit visual artifacts introduced in video frames by swapping algorithms.<n>This paper investigates the effectiveness of this approach by benchmarking CNN-based data-driven models on two data corpora.
  arXiv  Detail & Related papers  (2025-06-19T17:51:11Z)
• Prototypes are Balanced Units for Efficient and Effective Partially Relevant Video Retrieval [23.75587275795415]
  We propose a prototypical PRVR framework that encodes diverse contexts within a video into a fixed number of prototypes.<n>To keep the prototypes searchable via text queries while accurately encoding video contexts, we implement cross- and uni-modal reconstruction tasks.
  arXiv  Detail & Related papers  (2025-04-17T15:43:29Z)
• Facial Dynamics in Video: Instruction Tuning for Improved Facial Expression Perception and Contextual Awareness [6.634133253472436]
  This paper introduces a new instruction-following dataset tailored for dynamic facial expression caption.<n>The dataset comprises 5,033 high-quality video clips annotated manually, containing over 700,000 tokens.<n>We also present FEC-Bench, a benchmark designed to assess the performance of existing video MLLMs in this specific task.
  arXiv  Detail & Related papers  (2025-01-14T09:52:56Z)
• Reframe Anything: LLM Agent for Open World Video Reframing [0.8424099022563256]
  We introduce Reframe Any Video Agent (RAVA), an AI-based agent that restructures visual content for video reframing. RAVA operates in three stages: perception, where it interprets user instructions and video content; planning, where it determines aspect ratios and reframing strategies; and execution, where it invokes the editing tools to produce the final video. Our experiments validate the effectiveness of RAVA in video salient object detection and real-world reframing tasks, demonstrating its potential as a tool for AI-powered video editing.
  arXiv  Detail & Related papers  (2024-03-10T03:29:56Z)
• Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization [52.63845811751936]
  Video pre-training is challenging due to the modeling of its dynamics video. In this paper, we address such limitations in video pre-training with an efficient video decomposition. Our framework is both capable of comprehending and generating image and video content, as demonstrated by its performance across 13 multimodal benchmarks.
  arXiv  Detail & Related papers  (2024-02-05T16:30:49Z)
• EasyVolcap: Accelerating Neural Volumetric Video Research [69.59671164891725]
  Volumetric video is a technology that digitally records dynamic events such as artistic performances, sporting events, and remote conversations. EasyVolcap is a Python & Pytorch library for unifying the process of multi-view data processing, 4D scene reconstruction, and efficient dynamic volumetric video rendering.
  arXiv  Detail & Related papers  (2023-12-11T17:59:46Z)
• Video-based Person Re-identification with Long Short-Term Representation Learning [101.62570747820541]
  Video-based person Re-Identification (V-ReID) aims to retrieve specific persons from raw videos captured by non-overlapped cameras. We propose a novel deep learning framework named Long Short-Term Representation Learning (LSTRL) for effective V-ReID.
  arXiv  Detail & Related papers  (2023-08-07T16:22:47Z)
• Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models [149.1331903899298]
  We propose a novel framework called BIKE, which utilizes the cross-modal bridge to explore bidirectional knowledge. We present a Temporal Concept Spotting mechanism that uses the Text-to-Video expertise to capture temporal saliency in a parameter-free manner. Our best model achieves a state-of-the-art accuracy of 88.6% on the challenging Kinetics-400 using the released CLIP model.
  arXiv  Detail & Related papers  (2022-12-31T11:36:53Z)
• Facial Expression Video Generation Based-On Spatio-temporal Convolutional GAN: FEV-GAN [1.279257604152629]
  We present a novel approach for generating videos of the six basic facial expressions. Our approach is based on Spatio-temporal Conal GANs, that are known to model both content and motion in the same network. The code and the pre-trained model will soon be made publicly available.
  arXiv  Detail & Related papers  (2022-10-20T11:54:32Z)
• Audio-Visual Face Reenactment [34.79242760137663]
  This work proposes a novel method to generate realistic talking head videos using audio and visual streams. We animate a source image by transferring head motion from a driving video using a dense motion field generated using learnable keypoints. We improve the quality of lip sync using audio as an additional input, helping the network to attend to the mouth region.
  arXiv  Detail & Related papers  (2022-10-06T08:48:10Z)
• Multimodal Graph Learning for Deepfake Detection [10.077496841634135]
  Existing deepfake detectors face several challenges in achieving robustness and generalization. We propose a novel framework, namely Multimodal Graph Learning (MGL), that leverages information from multiple modalities. Our proposed method aims to effectively identify and utilize distinguishing features for deepfake detection.
  arXiv  Detail & Related papers  (2022-09-12T17:17:49Z)
• A Video Is Worth Three Views: Trigeminal Transformers for Video-based Person Re-identification [77.08204941207985]
  Video-based person re-identification (Re-ID) aims to retrieve video sequences of the same person under non-overlapping cameras. We propose a novel framework named Trigeminal Transformers (TMT) for video-based person Re-ID.
  arXiv  Detail & Related papers  (2021-04-05T02:50:16Z)
• Self-attention aggregation network for video face representation and recognition [0.0]
  We propose a new model architecture for video face representation and recognition based on a self-attention mechanism. Our approach could be used for video with single and multiple identities.
  arXiv  Detail & Related papers  (2020-10-11T20:57:46Z)
• An Emerging Coding Paradigm VCM: A Scalable Coding Approach Beyond Feature and Signal [99.49099501559652]
  Video Coding for Machine (VCM) aims to bridge the gap between visual feature compression and classical video coding. We employ a conditional deep generation network to reconstruct video frames with the guidance of learned motion pattern. By learning to extract sparse motion pattern via a predictive model, the network elegantly leverages the feature representation to generate the appearance of to-be-coded frames.
  arXiv  Detail & Related papers  (2020-01-09T14:18:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.