SQ-GAN: Semantic Image Communications Using Masked Vector Quantization

• URL: http://arxiv.org/abs/2502.09520v2
• Date: Fri, 10 Oct 2025 10:21:13 GMT
• Title: SQ-GAN: Semantic Image Communications Using Masked Vector Quantization
• Authors: Francesco Pezone, Sergio Barbarossa, Giuseppe Caire,
• Abstract summary: This work introduces Semantically Masked Vector Quantized Generative Adversarial Network (SQ-GAN)<n>It is a novel approach integrating semantically driven image coding and vector quantization to optimize image compression for semantic/task-oriented communications.<n>SQ-GAN outperforms state-of-the-art image compression schemes such as JPEG2000, BPG, and deep-learning based methods across multiple metrics.
• Score: 54.35918290143049
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This work introduces Semantically Masked Vector Quantized Generative Adversarial Network (SQ-GAN), a novel approach integrating semantically driven image coding and vector quantization to optimize image compression for semantic/task-oriented communications. The method only acts on source coding and is fully compliant with legacy systems. The semantics is extracted from the image computing its semantic segmentation map using off-the-shelf software. A new specifically developed semantic-conditioned adaptive mask module (SAMM) selectively encodes semantically relevant features of the image. The relevance of the different semantic classes is task-specific, and it is incorporated in the training phase by introducing appropriate weights in the loss function. SQ-GAN outperforms state-of-the-art image compression schemes such as JPEG2000, BPG, and deep-learning based methods across multiple metrics, including perceptual quality and semantic segmentation accuracy on the reconstructed image, at extremely low compression rates.

Related papers

• BPCLIP: A Bottom-up Image Quality Assessment from Distortion to Semantics Based on CLIP [18.25854559825818]
  We propose a bottom-up image quality assessment approach based on the Contrastive Language-Image Pre-training (CLIP)<n>Specifically, we utilize an encoder to extract multiscale features from the input image and introduce a bottom-up multiscale cross attention module.<n>By incorporating 40 image quality adjectives across six distinct dimensions, we enable the pre-trained CLIP text encoder to generate representations of the intrinsic quality of the image.
  arXiv  Detail & Related papers  (2025-06-22T09:56:57Z)
• Efficient Masked Image Compression with Position-Indexed Self-Attention [6.64044416324419]
  We propose an image compression method based on a position-indexed self-attention mechanism.<n>Compared to existing semantic-structured compression methods, our approach can significantly reduce computational costs.
  arXiv  Detail & Related papers  (2025-04-17T13:12:39Z)
• Extremely low-bitrate Image Compression Semantically Disentangled by LMMs from a Human Perception Perspective [2.542077227403488]
  Inspired by human progressive perception mechanism, we propose a Semantically Disentangled Image Compression framework. We leverage LMMs to extract essential semantic components, including overall descriptions, object detailed description, and semantic segmentation masks. We propose a training-free Object Restoration model with Attention Guidance (ORAG) built on pre-trained ControlNet to restore object details conditioned by object-level text descriptions and semantic masks.
  arXiv  Detail & Related papers  (2025-03-01T08:27:11Z)
• Hierarchical Semantic Compression for Consistent Image Semantic Restoration [62.97519327310638]
  We propose a novel hierarchical semantic compression (HSC) framework that purely operates within intrinsic semantic spaces from generative models. Experimental results demonstrate that the proposed HSC framework achieves the state-of-the-art performance on subjective quality and consistency for human vision.
  arXiv  Detail & Related papers  (2025-02-24T03:20:44Z)
• Semantics Prompting Data-Free Quantization for Low-Bit Vision Transformers [59.772673692679085]
  We propose SPDFQ, a Semantics Prompting Data-Free Quantization method for ViTs.<n>First, SPDFQ incorporates Attention Priors Alignment (APA), which uses randomly generated attention priors to enhance the semantics of synthetic images.<n>Second, SPDFQ introduces Multi-Semantic Reinforcement (MSR), which utilizes localized patch optimization to prompt efficient parameterization.
  arXiv  Detail & Related papers  (2024-12-21T09:30:45Z)
• Tell Codec What Worth Compressing: Semantically Disentangled Image Coding for Machine with LMMs [47.7670923159071]
  We present a new image compression paradigm to achieve intelligently coding for machine'' by cleverly leveraging the common sense of Large Multimodal Models (LMMs) We dub our method textitSDComp'' for textitSemantically textitDisentangled textitCompression'', and compare it with state-of-the-art codecs on a wide variety of different vision tasks.
  arXiv  Detail & Related papers  (2024-08-16T07:23:18Z)
• Image-GS: Content-Adaptive Image Representation via 2D Gaussians [52.598772767324036]
  We introduce Image-GS, a content-adaptive image representation based on 2D Gaussians radiance.<n>It supports hardware-friendly rapid access for real-time usage, requiring only 0.3K MACs to decode a pixel.<n>We demonstrate its versatility with several applications, including texture compression, semantics-aware compression, and joint image compression and restoration.
  arXiv  Detail & Related papers  (2024-07-02T00:45:21Z)
• Beyond Learned Metadata-based Raw Image Reconstruction [86.1667769209103]
  Raw images have distinct advantages over sRGB images, e.g., linearity and fine-grained quantization levels. They are not widely adopted by general users due to their substantial storage requirements. We propose a novel framework that learns a compact representation in the latent space, serving as metadata.
  arXiv  Detail & Related papers  (2023-06-21T06:59:07Z)
• Exploring Effective Mask Sampling Modeling for Neural Image Compression [171.35596121939238]
  Most existing neural image compression methods rely on side information from hyperprior or context models to eliminate spatial redundancy. Inspired by the mask sampling modeling in recent self-supervised learning methods for natural language processing and high-level vision, we propose a novel pretraining strategy for neural image compression. Our method achieves competitive performance with lower computational complexity compared to state-of-the-art image compression methods.
  arXiv  Detail & Related papers  (2023-06-09T06:50:20Z)
• SigVIC: Spatial Importance Guided Variable-Rate Image Compression [43.062173445454775]
  variable-rate mechanism has improved the flexibility and efficiency of learning-based image compression. One of the most common approaches for variable-rate is to channel-wisely or spatial-uniformly scale the internal features. We introduce a Spatial Importance Guided Variable-rate Image Compression (SigVIC), in which a spatial gating unit (SGU) is designed for adaptively learning a spatial importance mask.
  arXiv  Detail & Related papers  (2023-03-16T06:57:51Z)
• MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation [41.029441562130984]
  Two-stage Vector Quantized (VQ) generative models allow for synthesizing high-fidelity and high-resolution images. Our proposed modulated VQGAN is able to greatly improve the reconstructed image quality as well as provide high-fidelity image generation.
  arXiv  Detail & Related papers  (2022-09-19T13:26:51Z)
• Towards Semantic Communications: Deep Learning-Based Image Semantic Coding [42.453963827153856]
  We conceive the semantic communications for image data that is much more richer in semantics and bandwidth sensitive. We propose an reinforcement learning based adaptive semantic coding (RL-ASC) approach that encodes images beyond pixel level. Experimental results demonstrate that the proposed RL-ASC is noise robust and could reconstruct visually pleasant and semantic consistent image.
  arXiv  Detail & Related papers  (2022-08-08T12:29:55Z)
• Semantic Image Synthesis via Diffusion Models [159.4285444680301]
  Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks. Recent work on semantic image synthesis mainly follows the emphde facto Generative Adversarial Nets (GANs)
  arXiv  Detail & Related papers  (2022-06-30T18:31:51Z)
• Semi-parametric Makeup Transfer via Semantic-aware Correspondence [99.02329132102098]
  Large discrepancy between source non-makeup image and reference makeup image is one of key challenges in makeup transfer. Non-parametric techniques have a high potential for addressing the pose, expression, and occlusion discrepancies. We propose a textbfSemi-textbfparametric textbfMakeup textbfTransfer (SpMT) method, which combines the reciprocal strengths of non-parametric and parametric mechanisms.
  arXiv  Detail & Related papers  (2022-03-04T12:54:19Z)
• Transformer-based Image Compression [18.976159633970177]
  Transformer-based Image Compression (TIC) approach is developed which reuses the canonical variational autoencoder (VAE) architecture with paired main and hyper encoder-decoders. TIC rivals with state-of-the-art approaches including deep convolutional neural networks (CNNs) based learnt image coding (LIC) methods and handcrafted rules-based intra profile of recently-approved Versatile Video Coding (VVC) standard.
  arXiv  Detail & Related papers  (2021-11-12T13:13:20Z)
• Variable-Rate Deep Image Compression through Spatially-Adaptive Feature Transform [58.60004238261117]
  We propose a versatile deep image compression network based on Spatial Feature Transform (SFT arXiv:1804.02815) Our model covers a wide range of compression rates using a single model, which is controlled by arbitrary pixel-wise quality maps. The proposed framework allows us to perform task-aware image compressions for various tasks.
  arXiv  Detail & Related papers  (2021-08-21T17:30:06Z)
• Evidential fully convolutional network for semantic segmentation [6.230751621285322]
  We propose a hybrid architecture composed of a fully convolutional network (FCN) and a Dempster-Shafer layer for image semantic segmentation. Experiments show that the proposed combination improves the accuracy and calibration of semantic segmentation by assigning confusing pixels to multi-class sets.
  arXiv  Detail & Related papers  (2021-03-25T01:21:22Z)
• Hierarchical Quantized Autoencoders [3.9146761527401432]
  We motivate the use of a hierarchy of Vector Quantized Variencoders (VQ-VAEs) to attain high factors of compression. We show that a combination of quantization and hierarchical latent structure aids likelihood-based image compression. Our resulting scheme produces a Markovian series of latent variables that reconstruct images of high-perceptual quality.
  arXiv  Detail & Related papers  (2020-02-19T11:26:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.