Learning from Adversity: Semantic-Aware Mask Refinement through Adversarial Perturbation

• URL: http://arxiv.org/abs/2607.29059v1
• Date: Fri, 31 Jul 2026 06:26:37 GMT
• Title: Learning from Adversity: Semantic-Aware Mask Refinement through Adversarial Perturbation
• Abstract summary: We introduce Phoenix, a novel framework that leverages adversarial learning to generate semantically meaningful noise patterns.<n>Our approach consists of two key innovations: (1) Adrial Mask Perturbation, which employs embedding attacks to create semantic-aware noise that mimics real segmentation errors, and (2) Contrastive Mask Refinement Learning, which establishes a tri-directional framework that ensures feature consistency within semantic regions.
• Score: 56.11837114317144
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite significant advances in image segmentation, even state-of-the-art models produce masks with imperfect boundaries, semantic inconsistencies, and structural errors. Mask refinement addresses these limitations, yet current approaches rely on simplistic synthetic noise that fails to capture the complex error patterns of real segmentation models. We introduce Phoenix, a novel framework that leverages adversarial learning to generate semantically meaningful noise patterns and contrastive learning to model refinement relationships. Our approach consists of two key innovations: (1) Adversarial Mask Perturbation, which employs embedding attacks to create semantic-aware noise that mimics real segmentation errors, and (2) Contrastive Mask Refinement Learning, which establishes a tri-directional framework that ensures feature consistency within semantic regions while maintaining separation between classes. Experiments demonstrate that Phoenix significantly outperforms existing methods across diverse tasks, while consistently enhancing state-of-the-art segmentation models with substantial improvements. Our code and project page are publicly available at https://phoenix-eccv26.github.io.

Related papers

• Where Does Generative Difficulty Reside? An Empirical Study of Target Representations [62.54876287800644]
  The target representation defines the distribution an image generator must learn, yet it is often treated as an interchangeable interface.<n>We study raw pixels, SD-VAE latents and DINOv2 as well as MAE representation-autoencoder features within a unified masked autoregressive rectified-flow model.<n>Our results show that compression, reconstruction fidelity, token dimensionality, and visible semantic clustering do not individually predict generative behavior.
  arXiv  Detail & Related papers  (2026-08-01T12:32:31Z)
• Structured-Noise Masked Modeling for Video, Audio and Beyond [74.6545360752605]
  We introduce structured noise-based masking, a simple yet effective approach that naturally aligns with the spatial, temporal, and spectral characteristics of video and audio data.<n>Our approach improves the performance of masked video and audio modeling frameworks without any computational overhead.
  arXiv  Detail & Related papers  (2025-03-20T16:34:14Z)
• Masked Image Modeling Boosting Semi-Supervised Semantic Segmentation [38.55611683982936]
  We introduce a novel class-wise masked image modeling that independently reconstructs different image regions according to their respective classes. We develop a feature aggregation strategy that minimizes the distances between features corresponding to the masked and visible parts within the same class. In semantic space, we explore the application of masked image modeling to enhance regularization.
  arXiv  Detail & Related papers  (2024-11-13T16:42:07Z)
• Open-Vocabulary Segmentation with Unpaired Mask-Text Supervision [87.15580604023555]
  Unpair-Seg is a novel weakly-supervised open-vocabulary segmentation framework. It learns from unpaired image-mask and image-text pairs, which can be independently and efficiently collected. It achieves 14.6% and 19.5% mIoU on the ADE-847 and PASCAL Context-459 datasets.
  arXiv  Detail & Related papers  (2024-02-14T06:01:44Z)
• A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpainting [2.7563282688229664]
  This work builds upon Stable Diffusion and proposes a latent diffusion approach for panoptic segmentation. Our training consists of two steps: (1) training a shallow autoencoder to project the segmentation masks to latent space; (2) training a diffusion model to allow image-conditioned sampling in latent space.
  arXiv  Detail & Related papers  (2024-01-18T18:59:19Z)
• Mask2Anomaly: Mask Transformer for Universal Open-set Segmentation [29.43462426812185]
  We propose a paradigm change by shifting from a per-pixel classification to a mask classification. Our mask-based method, Mask2Anomaly, demonstrates the feasibility of integrating a mask-classification architecture. By comprehensive qualitative and qualitative evaluation, we show Mask2Anomaly achieves new state-of-the-art results.
  arXiv  Detail & Related papers  (2023-09-08T20:07:18Z)
• Denoising Diffusion Semantic Segmentation with Mask Prior Modeling [61.73352242029671]
  We propose to ameliorate the semantic segmentation quality of existing discriminative approaches with a mask prior modeled by a denoising diffusion generative model. We evaluate the proposed prior modeling with several off-the-shelf segmentors, and our experimental results on ADE20K and Cityscapes demonstrate that our approach could achieve competitively quantitative performance.
  arXiv  Detail & Related papers  (2023-06-02T17:47:01Z)
• Stare at What You See: Masked Image Modeling without Reconstruction [154.74533119863864]
  Masked Autoencoders (MAE) have been prevailing paradigms for large-scale vision representation pre-training. Recent approaches apply semantic-rich teacher models to extract image features as the reconstruction target, leading to better performance. We argue the features extracted by powerful teacher models already encode rich semantic correlation across regions in an intact image.
  arXiv  Detail & Related papers  (2022-11-16T12:48:52Z)
• ContrastMask: Contrastive Learning to Segment Every Thing [18.265503138997794]
  We propose ContrastMask, which learns a mask segmentation model on both seen and unseen categories. Features from the mask regions (foreground) are pulled together, and are contrasted against those from the background, and vice versa. Exhaustive experiments on the COCO dataset demonstrate the superiority of our method.
  arXiv  Detail & Related papers  (2022-03-18T07:41:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.