Object-Attribute-Relation Representation based Video Semantic Communication

• URL: http://arxiv.org/abs/2406.10469v1
• Date: Sat, 15 Jun 2024 02:19:31 GMT
• Title: Object-Attribute-Relation Representation based Video Semantic Communication
• Authors: Qiyuan Du, Yiping Duan, Qianqian Yang, Xiaoming Tao, Mérouane Debbah,
• Abstract summary: We introduce the use of object-attribute-relation (OAR) as a semantic framework for videos to facilitate low bit-rate coding. We utilize OAR sequences for both low bit-rate representation and generative video reconstruction. Our experiments on traffic surveillance video datasets assess the effectiveness of our approach in terms of video transmission performance.
• Score: 35.87160453583808
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rapid growth of multimedia data volume, there is an increasing need for efficient video transmission in applications such as virtual reality and future video streaming services. Semantic communication is emerging as a vital technique for ensuring efficient and reliable transmission in low-bandwidth, high-noise settings. However, most current approaches focus on joint source-channel coding (JSCC) that depends on end-to-end training. These methods often lack an interpretable semantic representation and struggle with adaptability to various downstream tasks. In this paper, we introduce the use of object-attribute-relation (OAR) as a semantic framework for videos to facilitate low bit-rate coding and enhance the JSCC process for more effective video transmission. We utilize OAR sequences for both low bit-rate representation and generative video reconstruction. Additionally, we incorporate OAR into the image JSCC model to prioritize communication resources for areas more critical to downstream tasks. Our experiments on traffic surveillance video datasets assess the effectiveness of our approach in terms of video transmission performance. The empirical findings demonstrate that our OAR-based video coding method not only outperforms H.265 coding at lower bit-rates but also synergizes with JSCC to deliver robust and efficient video transmission.

Related papers

• When Video Coding Meets Multimodal Large Language Models: A Unified Paradigm for Video Coding [112.44822009714461]
  Cross-Modality Video Coding (CMVC) is a pioneering approach to explore multimodality representation and video generative models in video coding. During decoding, previously encoded components and video generation models are leveraged to create multiple encoding-decoding modes. Experiments indicate that TT2V achieves effective semantic reconstruction, while IT2V exhibits competitive perceptual consistency.
  arXiv  Detail & Related papers  (2024-08-15T11:36:18Z)
• Text-Video Retrieval with Global-Local Semantic Consistent Learning [122.15339128463715]
  We propose a simple yet effective method, Global-Local Semantic Consistent Learning (GLSCL) GLSCL capitalizes on latent shared semantics across modalities for text-video retrieval. Our method achieves comparable performance with SOTA as well as being nearly 220 times faster in terms of computational cost.
  arXiv  Detail & Related papers  (2024-05-21T11:59:36Z)
• VideoQA-SC: Adaptive Semantic Communication for Video Question Answering [21.0279034601774]
  We propose an end-to-end SC system for video question answering tasks called VideoQA-SC. Our goal is to accomplish VideoQA tasks directly based on video semantics over noisy or fading wireless channels. Our results show the great potential of task-oriented SC system design for video applications.
  arXiv  Detail & Related papers  (2024-05-17T06:11:10Z)
• Cross-layer scheme for low latency multiple description video streaming over Vehicular Ad-hoc NETworks (VANETs) [2.2124180701409233]
  HEVC standard is very promising for real-time video streaming. New state-of-the-art video coding (HEVC) standard is very promising for real-time video streaming. We propose an original cross-layer system in order to enhance received video quality in vehicular communications.
  arXiv  Detail & Related papers  (2023-11-05T14:34:58Z)
• VNVC: A Versatile Neural Video Coding Framework for Efficient Human-Machine Vision [59.632286735304156]
  It is more efficient to enhance/analyze the coded representations directly without decoding them into pixels. We propose a versatile neural video coding (VNVC) framework, which targets learning compact representations to support both reconstruction and direct enhancement/analysis.
  arXiv  Detail & Related papers  (2023-06-19T03:04:57Z)
• Toward Adaptive Semantic Communications: Efficient Data Transmission via Online Learned Nonlinear Transform Source-Channel Coding [11.101344530143303]
  We propose an online learned joint source and channel coding approach that leverages the deep learning model's overfitting property. Specifically, we update the off-the-shelf pre-trained models after deployment in a lightweight online fashion to adapt to the distribution shifts in source data and environment domain. We take the overfitting concept to the extreme, proposing a series of implementation-friendly methods to adapt the model or representations to an individual data or channel state instance.
  arXiv  Detail & Related papers  (2022-11-08T16:00:27Z)
• Wireless Deep Video Semantic Transmission [14.071114007641313]
  We propose a new class of high-efficiency deep joint source-channel coding methods to achieve end-to-end video transmission over wireless channels. Our framework is collected under the name deep video semantic transmission (DVST)
  arXiv  Detail & Related papers  (2022-05-26T03:26:43Z)
• A Coding Framework and Benchmark towards Low-Bitrate Video Understanding [63.05385140193666]
  We propose a traditional-neural mixed coding framework that takes advantage of both traditional codecs and neural networks (NNs) The framework is optimized by ensuring that a transportation-efficient semantic representation of the video is preserved. We build a low-bitrate video understanding benchmark with three downstream tasks on eight datasets, demonstrating the notable superiority of our approach.
  arXiv  Detail & Related papers  (2022-02-06T16:29:15Z)
• Adaptive Intermediate Representations for Video Understanding [50.64187463941215]
  We introduce a new way to leverage semantic segmentation as an intermediate representation for video understanding. We propose a general framework which learns the intermediate representations (optical flow and semantic segmentation) jointly with the final video understanding task. We obtain more powerful visual representations for videos which lead to performance gains over the state-of-the-art.
  arXiv  Detail & Related papers  (2021-04-14T21:37:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.