Joint Transmission and Deblurring: A Semantic Communication Approach Using Events

• URL: http://arxiv.org/abs/2501.09396v1
• Date: Thu, 16 Jan 2025 09:07:01 GMT
• Title: Joint Transmission and Deblurring: A Semantic Communication Approach Using Events
• Authors: Pujing Yang, Guangyi Zhang, Yunlong Cai, Lei Yu, Guanding Yu,
• Abstract summary: Deep learning-based joint source-channel coding (JSCC) is emerging as a promising technology for effective image transmission. Most existing approaches focus on transmitting clear images, overlooking real-world challenges such as motion blur caused by camera shaking or fast-moving objects. We propose a novel J SCC framework for the joint transmission of blurry images and events, aimed at achieving high-quality reconstructions under limited channel bandwidth.
• Score: 30.60030333076227
• License:
• Abstract: Deep learning-based joint source-channel coding (JSCC) is emerging as a promising technology for effective image transmission. However, most existing approaches focus on transmitting clear images, overlooking real-world challenges such as motion blur caused by camera shaking or fast-moving objects. Motion blur often degrades image quality, making transmission and reconstruction more challenging. Event cameras, which asynchronously record pixel intensity changes with extremely low latency, have shown great potential for motion deblurring tasks. However, the efficient transmission of the abundant data generated by event cameras remains a significant challenge. In this work, we propose a novel JSCC framework for the joint transmission of blurry images and events, aimed at achieving high-quality reconstructions under limited channel bandwidth. This approach is designed as a deblurring task-oriented JSCC system. Since RGB cameras and event cameras capture the same scene through different modalities, their outputs contain both shared and domain-specific information. To avoid repeatedly transmitting the shared information, we extract and transmit their shared information and domain-specific information, respectively. At the receiver, the received signals are processed by a deblurring decoder to generate clear images. Additionally, we introduce a multi-stage training strategy to train the proposed model. Simulation results demonstrate that our method significantly outperforms existing JSCC-based image transmission schemes, addressing motion blur effectively.

Related papers

• Trustworthy Image Semantic Communication with GenAI: Explainablity, Controllability, and Efficiency [59.15544887307901]
  Image semantic communication (ISC) has garnered significant attention for its potential to achieve high efficiency in visual content transmission. Existing ISC systems based on joint source-channel coding face challenges in interpretability, operability, and compatibility. We propose a novel trustworthy ISC framework that employs Generative Artificial Intelligence (GenAI) for multiple downstream inference tasks.
  arXiv  Detail & Related papers  (2024-08-07T14:32:36Z)
• Object-Attribute-Relation Representation Based Video Semantic Communication [35.87160453583808]
  We introduce the use of object-attribute-relation (OAR) as a semantic framework for videos to facilitate low bit-rate coding. We utilize OAR sequences for both low bit-rate representation and generative video reconstruction. Our experiments on traffic surveillance video datasets assess the effectiveness of our approach in terms of video transmission performance.
  arXiv  Detail & Related papers  (2024-06-15T02:19:31Z)
• Neuromorphic Synergy for Video Binarization [54.195375576583864]
  Bimodal objects serve as a visual form to embed information that can be easily recognized by vision systems. Neuromorphic cameras offer new capabilities for alleviating motion blur, but it is non-trivial to first de-blur and then binarize the images in a real-time manner. We propose an event-based binary reconstruction method that leverages the prior knowledge of the bimodal target's properties to perform inference independently in both event space and image space. We also develop an efficient integration method to propagate this binary image to high frame rate binary video.
  arXiv  Detail & Related papers  (2024-02-20T01:43:51Z)
• Uncertainty-aware Bridge based Mobile-Former Network for Event-based Pattern Recognition [6.876867023375713]
  Event cameras perform better on high dynamic range, no motion blur, and low energy consumption. We propose a lightweight uncertainty-aware information propagation based Mobile-Former network for efficient pattern recognition.
  arXiv  Detail & Related papers  (2024-01-20T05:26:28Z)
• Learning Parallax for Stereo Event-based Motion Deblurring [8.201943408103995]
  Existing approaches rely on the perfect pixel-wise alignment between intensity images and events, which is not always fulfilled in the real world. We propose a novel coarse-to-fine framework, named NETwork of Event-based motion Deblurring with STereo event and intensity cameras (St-EDNet) We build a new dataset with STereo Event and Intensity Cameras (StEIC), containing real-world events, intensity images, and dense disparity maps.
  arXiv  Detail & Related papers  (2023-09-18T06:51:41Z)
• Alignment-free HDR Deghosting with Semantics Consistent Transformer [76.91669741684173]
  High dynamic range imaging aims to retrieve information from multiple low-dynamic range inputs to generate realistic output. Existing methods often focus on the spatial misalignment across input frames caused by the foreground and/or camera motion. We propose a novel alignment-free network with a Semantics Consistent Transformer (SCTNet) with both spatial and channel attention modules.
  arXiv  Detail & Related papers  (2023-05-29T15:03:23Z)
• MEFNet: Multi-scale Event Fusion Network for Motion Deblurring [62.60878284671317]
  Traditional frame-based cameras inevitably suffer from motion blur due to long exposure times. As a kind of bio-inspired camera, the event camera records the intensity changes in an asynchronous way with high temporal resolution. In this paper, we rethink the event-based image deblurring problem and unfold it into an end-to-end two-stage image restoration network.
  arXiv  Detail & Related papers  (2021-11-30T23:18:35Z)
• Bridging the Gap between Events and Frames through Unsupervised Domain Adaptation [57.22705137545853]
  We propose a task transfer method that allows models to be trained directly with labeled images and unlabeled event data. We leverage the generative event model to split event features into content and motion features. Our approach unlocks the vast amount of existing image datasets for the training of event-based neural networks.
  arXiv  Detail & Related papers  (2021-09-06T17:31:37Z)
• Learning Event-Based Motion Deblurring [39.16921854492941]
  Fast motion can be captured as events at high time rate for event-based cameras. We show how its optimization can be unfolded with a novel end-to-end deep architecture. The proposed approach achieves state-of-the-art reconstruction quality, and generalizes better to handling real-world motion blur.
  arXiv  Detail & Related papers  (2020-04-13T07:01:06Z)
• EventSR: From Asynchronous Events to Image Reconstruction, Restoration, and Super-Resolution via End-to-End Adversarial Learning [75.17497166510083]
  Event cameras sense intensity changes and have many advantages over conventional cameras. Some methods have been proposed to reconstruct intensity images from event streams. The outputs are still in low resolution (LR), noisy, and unrealistic. We propose a novel end-to-end pipeline that reconstructs LR images from event streams, enhances the image qualities and upsamples the enhanced images, called EventSR.
  arXiv  Detail & Related papers  (2020-03-17T10:58:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.