JPD-SE: High-Level Semantics for Joint Perception-Distortion Enhancement in Image Compression

• URL: http://arxiv.org/abs/2005.12810v3
• Date: Tue, 9 Aug 2022 05:53:12 GMT
• Title: JPD-SE: High-Level Semantics for Joint Perception-Distortion Enhancement in Image Compression
• Abstract summary: We present a generic framework that can enable any image to leverage high-level semantics and study the joint optimization of perception quality and distortion. We propose a three-phase training scheme that teaches semantic-aware codecs to leverage the power of semantic to jointly optimize rate-perception-distortion (R-PD) performance.
• Score: 11.668382772425913
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While humans can effortlessly transform complex visual scenes into simple words and the other way around by leveraging their high-level understanding of the content, conventional or the more recent learned image compression codecs do not seem to utilize the semantic meanings of visual content to their full potential. Moreover, they focus mostly on rate-distortion and tend to underperform in perception quality especially in low bitrate regime, and often disregard the performance of downstream computer vision algorithms, which is a fast-growing consumer group of compressed images in addition to human viewers. In this paper, we (1) present a generic framework that can enable any image codec to leverage high-level semantics and (2) study the joint optimization of perception quality and distortion. Our idea is that given any codec, we utilize high-level semantics to augment the low-level visual features extracted by it and produce essentially a new, semantic-aware codec. We propose a three-phase training scheme that teaches semantic-aware codecs to leverage the power of semantic to jointly optimize rate-perception-distortion (R-PD) performance. As an additional benefit, semantic-aware codecs also boost the performance of downstream computer vision algorithms. To validate our claim, we perform extensive empirical evaluations and provide both quantitative and qualitative results.

Related papers

• CodecArena: Codec Quality Assessment via Visual Reinforcement Learning [70.59130074737446]
  We propose CodecArena, the first vision-language framework for video coding quality assessment.<n>CodecArena casts evaluation as source-conditioned comparative reasoning between a reference and its reconstructions.<n>Its facet-anchored reward uses automatically derived facet directions as weak anchors, rather than human per-facet labels.
  arXiv  Detail & Related papers  (2026-08-10T05:37:06Z)
• A Projection-Based Surrogate Gradient Interpretation for Neural Codec Wrappers [9.436544348188598]
  We show that a surrogate gradient can be interpreted as a first-order local approximation of the video.<n>We show that it is effective not only for learning downscaling operations, but also for the more challenging task of full neural wrapping.
  arXiv  Detail & Related papers  (2026-06-12T11:29:17Z)
• Coarse-to-Fine: Progressive Image Compression for Semantically Hierarchical Classification [27.208988763458958]
  We introduce a semantic hierarchy-aware progressive codecs that enables semantic scalability (i.e., coarse-to-fine) from a single bitstream.<n>Our work provides an efficient and interpretable solution for task-adaptive image coding, outperforming existing progressive hierarchical evaluation.
  arXiv  Detail & Related papers  (2026-05-08T02:08:35Z)
• SCENE: Semantic-aware Codec Enhancement with Neural Embeddings [9.6489283335586]
  We propose a semantic-aware pre-processing framework that enhances perceptual fidelity.<n>Our method integrates semantic embeddings from a vision-language model into an efficient convolutional architecture.<n>Our results show that semantic-aware pre-processing is an effective approach for enhancing compressed video streams.
  arXiv  Detail & Related papers  (2026-01-29T05:41:28Z)
• Progressive Learned Image Compression for Machine Perception [27.208988763458958]
  We propose a novel progressive learned image compression for machine perception, PICM-Net, based on trit-plane coding.<n>Our approach enables efficient and adaptive progressive transmission while maintaining high performance in the downstream classification task.
  arXiv  Detail & Related papers  (2025-12-23T05:45:38Z)
• Extremely low-bitrate Image Compression Semantically Disentangled by LMMs from a Human Perception Perspective [2.542077227403488]
  Inspired by human progressive perception mechanism, we propose a Semantically Disentangled Image Compression framework. We leverage LMMs to extract essential semantic components, including overall descriptions, object detailed description, and semantic segmentation masks. We propose a training-free Object Restoration model with Attention Guidance (ORAG) built on pre-trained ControlNet to restore object details conditioned by object-level text descriptions and semantic masks.
  arXiv  Detail & Related papers  (2025-03-01T08:27:11Z)
• Hierarchical Semantic Compression for Consistent Image Semantic Restoration [62.97519327310638]
  We propose a novel hierarchical semantic compression (HSC) framework that purely operates within intrinsic semantic spaces from generative models. Experimental results demonstrate that the proposed HSC framework achieves the state-of-the-art performance on subjective quality and consistency for human vision.
  arXiv  Detail & Related papers  (2025-02-24T03:20:44Z)
• Tell Codec What Worth Compressing: Semantically Disentangled Image Coding for Machine with LMMs [47.7670923159071]
  We present a new image compression paradigm to achieve intelligently coding for machine'' by cleverly leveraging the common sense of Large Multimodal Models (LMMs) We dub our method textitSDComp'' for textitSemantically textitDisentangled textitCompression'', and compare it with state-of-the-art codecs on a wide variety of different vision tasks.
  arXiv  Detail & Related papers  (2024-08-16T07:23:18Z)
• Compression-Realized Deep Structural Network for Video Quality Enhancement [78.13020206633524]
  This paper focuses on the task of quality enhancement for compressed videos. Most of the existing methods lack a structured design to optimally leverage the priors within compression codecs. A new paradigm is urgently needed for a more conscious'' process of quality enhancement.
  arXiv  Detail & Related papers  (2024-05-10T09:18:17Z)
• Semantic Ensemble Loss and Latent Refinement for High-Fidelity Neural Image Compression [58.618625678054826]
  This study presents an enhanced neural compression method designed for optimal visual fidelity. We have trained our model with a sophisticated semantic ensemble loss, integrating Charbonnier loss, perceptual loss, style loss, and a non-binary adversarial loss. Our empirical findings demonstrate that this approach significantly improves the statistical fidelity of neural image compression.
  arXiv  Detail & Related papers  (2024-01-25T08:11:27Z)
• VNVC: A Versatile Neural Video Coding Framework for Efficient Human-Machine Vision [59.632286735304156]
  It is more efficient to enhance/analyze the coded representations directly without decoding them into pixels. We propose a versatile neural video coding (VNVC) framework, which targets learning compact representations to support both reconstruction and direct enhancement/analysis.
  arXiv  Detail & Related papers  (2023-06-19T03:04:57Z)
• Towards Semantic Communications: Deep Learning-Based Image Semantic Coding [42.453963827153856]
  We conceive the semantic communications for image data that is much more richer in semantics and bandwidth sensitive. We propose an reinforcement learning based adaptive semantic coding (RL-ASC) approach that encodes images beyond pixel level. Experimental results demonstrate that the proposed RL-ASC is noise robust and could reconstruct visually pleasant and semantic consistent image.
  arXiv  Detail & Related papers  (2022-08-08T12:29:55Z)
• Preprocessing Enhanced Image Compression for Machine Vision [14.895698385236937]
  We propose a preprocessing enhanced image compression method for machine vision tasks. Our framework is built upon the traditional non-differential codecs. Experimental results show our method achieves a better tradeoff between the coding and the performance of the downstream machine vision tasks by saving about 20%.
  arXiv  Detail & Related papers  (2022-06-12T03:36:38Z)
• In-N-Out Generative Learning for Dense Unsupervised Video Segmentation [89.21483504654282]
  In this paper, we focus on the unsupervised Video Object (VOS) task which learns visual correspondence from unlabeled videos. We propose the In-aNd-Out (INO) generative learning from a purely generative perspective, which captures both high-level and fine-grained semantics. Our INO outperforms previous state-of-the-art methods by significant margins.
  arXiv  Detail & Related papers  (2022-03-29T07:56:21Z)
• Neural JPEG: End-to-End Image Compression Leveraging a Standard JPEG Encoder-Decoder [73.48927855855219]
  We propose a system that learns to improve the encoding performance by enhancing its internal neural representations on both the encoder and decoder ends. Experiments demonstrate that our approach successfully improves the rate-distortion performance over JPEG across various quality metrics.
  arXiv  Detail & Related papers  (2022-01-27T20:20:03Z)
• Learned Image Compression for Machine Perception [17.40776913809306]
  We develop a framework that produces a compression format suitable for both human perception and machine perception. We show that representations can be learned that simultaneously optimize for compression and performance on core vision tasks.
  arXiv  Detail & Related papers  (2021-11-03T14:39:09Z)
• How to Exploit the Transferability of Learned Image Compression to Conventional Codecs [25.622863999901874]
  We show how learned image coding can be used as a surrogate to optimize an image for encoding. Our approach can remodel a conventional image to adjust for the MS-SSIM distortion with over 20% rate improvement without any decoding overhead.
  arXiv  Detail & Related papers  (2020-12-03T12:34:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.