Cityscape-Adverse: Benchmarking Robustness of Semantic Segmentation with Realistic Scene Modifications via Diffusion-Based Image Editing

• URL: http://arxiv.org/abs/2411.00425v1
• Date: Fri, 01 Nov 2024 07:47:37 GMT
• Title: Cityscape-Adverse: Benchmarking Robustness of Semantic Segmentation with Realistic Scene Modifications via Diffusion-Based Image Editing
• Abstract summary: We introduce Cityscape-Adverse, a benchmark that employs diffusion-based image editing to simulate eight adverse conditions. We evaluate the reliability of diffusion-based models in generating realistic scene modifications and assess the performance of CNN and Transformer-based semantic segmentation models. We verify that models trained on Cityscape-Adverse show significantly enhanced resilience when applied to unseen domains.
• Score: 1.6316196319061707
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements in generative AI, particularly diffusion-based image editing, have enabled the transformation of images into highly realistic scenes using only text instructions. This technology offers significant potential for generating diverse synthetic datasets to evaluate model robustness. In this paper, we introduce Cityscape-Adverse, a benchmark that employs diffusion-based image editing to simulate eight adverse conditions, including variations in weather, lighting, and seasons, while preserving the original semantic labels. We evaluate the reliability of diffusion-based models in generating realistic scene modifications and assess the performance of state-of-the-art CNN and Transformer-based semantic segmentation models under these challenging conditions. Additionally, we analyze which modifications have the greatest impact on model performance and explore how training on synthetic datasets can improve robustness in real-world adverse scenarios. Our results demonstrate that all tested models, particularly CNN-based architectures, experienced significant performance degradation under extreme conditions, while Transformer-based models exhibited greater resilience. We verify that models trained on Cityscape-Adverse show significantly enhanced resilience when applied to unseen domains. Code and datasets will be released at https://github.com/naufalso/cityscape-adverse.

Related papers

• Texture-Shape Bias Balancing for Robust Synthetic-to-Real Semantic Segmentation in Automotive NIR Imagery [6.537821116235228]
  We present the first systematic study on synthetic to real domain adaptation for semantic segmentation in NIR images in the automotive domain.<n>We propose a generative augmentation framework that transforms synthetic images into realistic NIR-style variants.<n>Experiments show that balancing inductive bias during training leads to noticeably more robust semantic segmentation.
  arXiv  Detail & Related papers  (2026-06-13T03:01:10Z)
• Edit2Perceive: Image Editing Diffusion Models Are Strong Dense Perceivers [55.15722080205737]
  Edit2Perceive is a unified diffusion framework that adapts editing models for depth, normal, and matting.<n>Our single-step deterministic inference yields up to faster runtime while training on relatively small datasets.
  arXiv  Detail & Related papers  (2025-11-24T01:13:51Z)
• Scaling Transformer-Based Novel View Synthesis Models with Token Disentanglement and Synthetic Data [53.040873127309766]
  We propose a token disentanglement process within the transformer architecture, enhancing feature separation and ensuring more effective learning.<n>Our method outperforms existing models on both in-dataset and cross-dataset evaluations.
  arXiv  Detail & Related papers  (2025-09-08T17:58:06Z)
• TDiR: Transformer based Diffusion for Image Restoration Tasks [19.992144590243836]
  Images captured in challenging environments often experience various forms of degradation, including noise, color cast, blur, and light scattering.<n>These effects significantly reduce image quality, hindering their applicability in downstream tasks such as object detection, mapping, and classification.<n>Our transformer-based diffusion model was developed to address image restoration tasks, aiming to improve the quality of degraded images.
  arXiv  Detail & Related papers  (2025-06-25T10:28:13Z)
• Digi2Real: Bridging the Realism Gap in Synthetic Data Face Recognition via Foundation Models [4.910937238451485]
  We introduce a novel framework for realism transfer aimed at enhancing the realism of synthetically generated face images. By integrating the controllable aspects of the graphics pipeline with our realism enhancement technique, we generate a large amount of realistic variations.
  arXiv  Detail & Related papers  (2024-11-04T15:42:22Z)
• Data Augmentation via Latent Diffusion for Saliency Prediction [67.88936624546076]
  Saliency prediction models are constrained by the limited diversity and quantity of labeled data. We propose a novel data augmentation method for deep saliency prediction that edits natural images while preserving the complexity and variability of real-world scenes.
  arXiv  Detail & Related papers  (2024-09-11T14:36:24Z)
• Diffusion Model Driven Test-Time Image Adaptation for Robust Skin Lesion Classification [24.08402880603475]
  We propose a test-time image adaptation method to enhance the accuracy of the model on test data. We modify the target test images by projecting them back to the source domain using a diffusion model. Our method makes the robustness more robust across various corruptions, architectures, and data regimes.
  arXiv  Detail & Related papers  (2024-05-18T13:28:51Z)
• Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization [62.157627519792946]
  We introduce a novel framework called bridged transfer, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability. We propose dataset style inversion strategy to improve the stylistic alignment between synthetic and real images. Our proposed methods are evaluated across 10 different datasets and 5 distinct models, demonstrating consistent improvements.
  arXiv  Detail & Related papers  (2024-03-28T22:25:05Z)
• ObjectCompose: Evaluating Resilience of Vision-Based Models on Object-to-Background Compositional Changes [64.57705752579207]
  We evaluate the resilience of vision-based models against diverse object-to-background context variations. We harness the generative capabilities of text-to-image, image-to-text, and image-to-segment models to automatically generate object-to-background changes.
  arXiv  Detail & Related papers  (2024-03-07T17:48:48Z)
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis [22.11487736315616]
  Rectified flow is a recent generative model formulation that connects data and noise in a straight line. We improve existing noise sampling techniques for training rectified flow models by biasing them towards perceptually relevant scales. We present a novel transformer-based architecture for text-to-image generation that uses separate weights for the two modalities.
  arXiv  Detail & Related papers  (2024-03-05T18:45:39Z)
• DGNet: Dynamic Gradient-Guided Network for Water-Related Optics Image Enhancement [77.0360085530701]
  Underwater image enhancement (UIE) is a challenging task due to the complex degradation caused by underwater environments. Previous methods often idealize the degradation process, and neglect the impact of medium noise and object motion on the distribution of image features. Our approach utilizes predicted images to dynamically update pseudo-labels, adding a dynamic gradient to optimize the network's gradient space.
  arXiv  Detail & Related papers  (2023-12-12T06:07:21Z)
• Improving the Effectiveness of Deep Generative Data [5.856292656853396]
  Training a model on purely synthetic images for downstream image processing tasks results in an undesired performance drop compared to training on real data. We propose a new taxonomy to describe factors contributing to this commonly observed phenomenon and investigate it on the popular CIFAR-10 dataset. Our method outperforms baselines on downstream classification tasks both in case of training on synthetic only (Synthetic-to-Real) and training on a mix of real and synthetic data.
  arXiv  Detail & Related papers  (2023-11-07T12:57:58Z)
• Transferring and Regularizing Prediction for Semantic Segmentation [115.88957139226966]
  In this paper, we exploit the intrinsic properties of semantic segmentation to alleviate such problem for model transfer. We present a Regularizer of Prediction Transfer (RPT) that imposes the intrinsic properties as constraints to regularize model transfer in an unsupervised fashion. Extensive experiments are conducted to verify the proposal of RPT on the transfer of models trained on GTA5 and SYNTHIA (synthetic data) to Cityscapes dataset (urban street scenes)
  arXiv  Detail & Related papers  (2020-06-11T16:19:41Z)
• Deep CG2Real: Synthetic-to-Real Translation via Image Disentanglement [78.58603635621591]
  Training an unpaired synthetic-to-real translation network in image space is severely under-constrained. We propose a semi-supervised approach that operates on the disentangled shading and albedo layers of the image. Our two-stage pipeline first learns to predict accurate shading in a supervised fashion using physically-based renderings as targets.
  arXiv  Detail & Related papers  (2020-03-27T21:45:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.