Beyond Fidelity: Semantic Similarity Assessment in Low-Level Image Processing

• URL: http://arxiv.org/abs/2604.25408v1
• Date: Tue, 28 Apr 2026 09:21:05 GMT
• Title: Beyond Fidelity: Semantic Similarity Assessment in Low-Level Image Processing
• Abstract summary: We formalize textitSemantic Similarity as a new evaluation task for low-level image processing.<n>We present a structured formulation of image semantics based on semantic entities and their relations.<n>We propose Triplet-based Semantic Similarity Score (T3S), which models image semantics through foreground entities, background entities, and relations.
• Score: 53.27421114481586
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Low-level image processing has long been evaluated mainly from the perspective of visual fidelity. However, with the rise of deep learning and generative models, processed images may preserve perceptual quality while altering semantic content, making conventional Image Quality Assessment (IQA) insufficient for semantic-level assessment. In this paper, we formalize \textit{Semantic Similarity} as a new evaluation task for low-level image processing, aimed at measuring whether semantic content is preserved after processing. We further present a structured formulation of image semantics based on semantic entities and their relations, and discuss the desired properties and constraints of a valid semantic similarity index. Based on this formulation, we propose Triplet-based Semantic Similarity Score (T3S), which models image semantics through foreground entities, background entities, and relations. T3S combines semantic entity extraction, foreground-background disentanglement, and open-world class/relation modeling. Experiments on COCO and SPA-Data show that T3S consistently outperforms existing fidelity-oriented metrics and representative semantic-level baselines, while better reflecting progressive semantic changes under diverse degradations. These results highlight the importance of semantic assessment in modern low-level vision.

Related papers

• BPCLIP: A Bottom-up Image Quality Assessment from Distortion to Semantics Based on CLIP [18.25854559825818]
  We propose a bottom-up image quality assessment approach based on the Contrastive Language-Image Pre-training (CLIP)<n>Specifically, we utilize an encoder to extract multiscale features from the input image and introduce a bottom-up multiscale cross attention module.<n>By incorporating 40 image quality adjectives across six distinct dimensions, we enable the pre-trained CLIP text encoder to generate representations of the intrinsic quality of the image.
  arXiv  Detail & Related papers  (2025-06-22T09:56:57Z)
• Introspective Deep Metric Learning [91.47907685364036]
  We propose an introspective deep metric learning framework for uncertainty-aware comparisons of images. The proposed IDML framework improves the performance of deep metric learning through uncertainty modeling.
  arXiv  Detail & Related papers  (2023-09-11T16:21:13Z)
• Edge Guided GANs with Multi-Scale Contrastive Learning for Semantic Image Synthesis [139.2216271759332]
  We propose a novel ECGAN for the challenging semantic image synthesis task. The semantic labels do not provide detailed structural information, making it challenging to synthesize local details and structures. The widely adopted CNN operations such as convolution, down-sampling, and normalization usually cause spatial resolution loss. We propose a novel contrastive learning method, which aims to enforce pixel embeddings belonging to the same semantic class to generate more similar image content.
  arXiv  Detail & Related papers  (2023-07-22T14:17:19Z)
• Regression-free Blind Image Quality Assessment with Content-Distortion Consistency [42.683300312253884]
  We propose a regression-free framework for image quality evaluation. It is based upon retrieving locally similar instances by incorporating semantic and distortion feature spaces. The proposed method achieves competitive, even superior performance compared to state-of-the-art regression-based methods.
  arXiv  Detail & Related papers  (2023-07-18T14:19:28Z)
• LSEH: Semantically Enhanced Hard Negatives for Cross-modal Information Retrieval [0.4264192013842096]
  Visual Semantic Embedding (VSE) aims to extract the semantics of images and their descriptions, and embed them into the same latent space for information retrieval. Most existing VSE networks are trained by adopting a hard negatives loss function which learns an objective margin between the similarity of relevant and irrelevant image-description embedding pairs. This paper presents a novel approach that comprises two main parts: (1) finds the underlying semantics of image descriptions; and (2) proposes a novel semantically enhanced hard negatives loss function.
  arXiv  Detail & Related papers  (2022-10-10T15:09:39Z)
• Towards Semantic Communications: Deep Learning-Based Image Semantic Coding [42.453963827153856]
  We conceive the semantic communications for image data that is much more richer in semantics and bandwidth sensitive. We propose an reinforcement learning based adaptive semantic coding (RL-ASC) approach that encodes images beyond pixel level. Experimental results demonstrate that the proposed RL-ASC is noise robust and could reconstruct visually pleasant and semantic consistent image.
  arXiv  Detail & Related papers  (2022-08-08T12:29:55Z)
• Semantic Image Synthesis via Diffusion Models [174.24523061460704]
  Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks.<n>Recent work on semantic image synthesis mainly follows the de facto GAN-based approaches.<n>We propose a novel framework based on DDPM for semantic image synthesis.
  arXiv  Detail & Related papers  (2022-06-30T18:31:51Z)
• Introspective Deep Metric Learning for Image Retrieval [80.29866561553483]
  We argue that a good similarity model should consider the semantic discrepancies with caution to better deal with ambiguous images for more robust training. We propose to represent an image using not only a semantic embedding but also an accompanying uncertainty embedding, which describes the semantic characteristics and ambiguity of an image, respectively. The proposed IDML framework improves the performance of deep metric learning through uncertainty modeling and attains state-of-the-art results on the widely used CUB-200-2011, Cars196, and Stanford Online Products datasets.
  arXiv  Detail & Related papers  (2022-05-09T17:51:44Z)
• Retrieval-based Spatially Adaptive Normalization for Semantic Image Synthesis [68.1281982092765]
  We propose a novel normalization module, termed as REtrieval-based Spatially AdaptIve normaLization (RESAIL) RESAIL provides pixel level fine-grained guidance to the normalization architecture. Experiments on several challenging datasets show that our RESAIL performs favorably against state-of-the-arts in terms of quantitative metrics, visual quality, and subjective evaluation.
  arXiv  Detail & Related papers  (2022-04-06T14:21:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.