Scalable, Energy-Efficient Optical-Neural Architecture for Multiplexed Deepfake Video Detection

• URL: http://arxiv.org/abs/2605.19360v1
• Date: Tue, 19 May 2026 04:42:47 GMT
• Title: Scalable, Energy-Efficient Optical-Neural Architecture for Multiplexed Deepfake Video Detection
• Abstract summary: We present a hybrid digital- analog deepfake video detection framework that combines a lightweight digital front-end with a spatially multiplexed optical decoding back-end.<n>We achieve average deepfake detection accuracy, sensitivity and specificity of 97.79%, 99.86% and 95.72%, respectively, on the Celeb-DF video dataset.<n>Our results show that integrating optical computation into AI inference enables simultaneous gains in throughput, energy efficiency, and adversarial robustness.
• Score: 7.349238139625144
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid proliferation of AI-generated visual media has created an urgent need for efficient, trustworthy deepfake detection systems. However, existing deep learning-based detection methods rely on computationally intensive and energy-demanding inference algorithms, limiting their scalability. Here, we present a hybrid digital-analog deepfake video detection framework that combines a lightweight digital front-end with a spatially multiplexed optical decoding back-end for massively parallel analog inference through a programmable spatial light modulator. By simultaneously processing 15 or more video streams within a single optical propagation pass, the system enables high-throughput and accurate video-level authenticity prediction at reduced computational cost compared with purely digital methods. We validated this hybrid deepfake video processor using different datasets spanning classical face-swapping, real-world deepfake recordings, and fully AI-generated videos. Using a spatially multiplexed experimental set-up operating in the visible spectrum, we achieved average deepfake detection accuracy, sensitivity and specificity of 97.79%, 99.86% and 95.72%, respectively, on the Celeb-DF video dataset with 15 videos tested in parallel in a single optical pass per inference. The multiplexed optical decoder also demonstrates resilience against various types of video degradation, noise, compression, experimental misalignments and black-box adversarial attacks. Our results show that integrating optical computation into AI inference enables simultaneous gains in throughput, energy efficiency, and adversarial robustness - three properties that are difficult to achieve together in purely digital systems.

Related papers

• AccVideo: Accelerating Video Diffusion Model with Synthetic Dataset [55.82208863521353]
  We propose AccVideo to reduce the inference steps for accelerating video diffusion models with synthetic dataset.<n>Our model achieves 8.5x improvements in generation speed compared to the teacher model.<n>Compared to previous accelerating methods, our approach is capable of generating videos with higher quality and resolution.
  arXiv  Detail & Related papers  (2025-03-25T08:52:07Z)
• Optical-Flow Guided Prompt Optimization for Coherent Video Generation [51.430833518070145]
  We propose a framework called MotionPrompt that guides the video generation process via optical flow.<n>We optimize learnable token embeddings during reverse sampling steps by using gradients from a trained discriminator applied to random frame pairs.<n>This approach allows our method to generate visually coherent video sequences that closely reflect natural motion dynamics, without compromising the fidelity of the generated content.
  arXiv  Detail & Related papers  (2024-11-23T12:26:52Z)
• Transferable polychromatic optical encoder for neural networks [13.311727599288524]
  In this paper, we demonstrate an optical encoder that can perform convolution simultaneously in three color channels during the image capture. Such an optical encoding results in 24,000 times reduction in computational operations, with a state-of-the art classification accuracy (73.2%) in free-space optical system.
  arXiv  Detail & Related papers  (2024-11-05T00:49:47Z)
• BVI-RLV: A Fully Registered Dataset and Benchmarks for Low-Light Video Enhancement [56.97766265018334]
  This paper introduces a low-light video dataset, consisting of 40 scenes with various motion scenarios under two distinct low-lighting conditions. We provide fully registered ground truth data captured in normal light using a programmable motorized dolly and refine it via an image-based approach for pixel-wise frame alignment across different light levels. Our experimental results demonstrate the significance of fully registered video pairs for low-light video enhancement (LLVE) and the comprehensive evaluation shows that the models trained with our dataset outperform those trained with the existing datasets.
  arXiv  Detail & Related papers  (2024-07-03T22:41:49Z)
• Deep Optics for Video Snapshot Compressive Imaging [10.830072985735175]
  Video snapshot imaging (SCI) aims to capture a sequence of video frames with only a single shot of a 2D detector. This paper presents a framework to jointly optimize masks and a reconstruction network. We believe this is a milestone for real-world video SCI.
  arXiv  Detail & Related papers  (2024-04-08T08:04:44Z)
• Digital-analog hybrid matrix multiplication processor for optical neural networks [11.171425574890765]
  We propose a digital-analog hybrid optical computing architecture for optical neural networks (ONNs) By introducing the logic levels and decisions based on thresholding, the calculation precision can be significantly enhanced. We have demonstrated an unprecedented 16-bit calculation precision for high-definition image processing, with a pixel error rate (PER) as low as $1.8times10-3$ at a signal-to-noise ratio (SNR) of 18.2 dB.
  arXiv  Detail & Related papers  (2024-01-26T18:42:57Z)
• Spatiotemporal Attention-based Semantic Compression for Real-time Video Recognition [117.98023585449808]
  We propose a temporal attention-based autoencoder (STAE) architecture to evaluate the importance of frames and pixels in each frame. We develop a lightweight decoder that leverages a 3D-2D CNN combined to reconstruct missing information. Experimental results show that ViT_STAE can compress the video dataset H51 by 104x with only 5% accuracy loss.
  arXiv  Detail & Related papers  (2023-05-22T07:47:27Z)
• TMM-Fast: A Transfer Matrix Computation Package for Multilayer Thin-Film Optimization [62.997667081978825]
  An advanced thin-film structure can consist of multiple materials with different thicknesses and numerous layers. Design and optimization of complex thin-film structures with multiple variables is a computationally heavy problem that is still under active research. We propose the Python package TMM-Fast which enables parallelized computation of reflection and transmission of light at different angles of incidence and wavelengths through the multilayer thin-film.
  arXiv  Detail & Related papers  (2021-11-24T14:47:37Z)
• Dual-view Snapshot Compressive Imaging via Optical Flow Aided Recurrent Neural Network [14.796204921975733]
  Dual-view snapshot compressive imaging (SCI) aims to capture videos from two field-of-views (FoVs) in a single snapshot. It is challenging for existing model-based decoding algorithms to reconstruct each individual scene. We propose an optical flow-aided recurrent neural network for dual video SCI systems, which provides high-quality decoding in seconds.
  arXiv  Detail & Related papers  (2021-09-11T14:24:44Z)
• M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection [74.19291916812921]
  forged images generated by Deepfake techniques pose a serious threat to the trustworthiness of digital information. In this paper, we aim to capture the subtle manipulation artifacts at different scales for Deepfake detection. We introduce a high-quality Deepfake dataset, SR-DF, which consists of 4,000 DeepFake videos generated by state-of-the-art face swapping and facial reenactment methods.
  arXiv  Detail & Related papers  (2021-04-20T05:43:44Z)
• Scalable Optical Learning Operator [0.2399911126932526]
  The presented framework overcomes the energy scaling problem of existing systems without classifying speed. We numerically and experimentally showed the ability of the method to execute several different tasks with accuracy comparable to a digital implementation. Our results indicate that a powerful supercomputer would be required to duplicate the performance of the multimode fiber-based computer.
  arXiv  Detail & Related papers  (2020-12-22T23:06:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.