DA-STC: Domain Adaptive Video Semantic Segmentation via Spatio-Temporal Consistency

• URL: http://arxiv.org/abs/2311.13254v1
• Date: Wed, 22 Nov 2023 09:18:49 GMT
• Title: DA-STC: Domain Adaptive Video Semantic Segmentation via Spatio-Temporal Consistency
• Authors: Zhe Zhang, Gaochang Wu, Jing Zhang, Chunhua Shen, Dacheng Tao, Tianyou Chai
• Abstract summary: We propose a DA-STC method for domain adaptive video semantic segmentation, which incorporates a multi-level-level fusion module. We also propose a category-aware feature alignment module to promote the consistency of significant-temporal features. Our method exhibits superior performance for domain adaptive semantic segmentation.
• Score: 111.48756648371256
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video semantic segmentation is a pivotal aspect of video representation learning. However, significant domain shifts present a challenge in effectively learning invariant spatio-temporal features across the labeled source domain and unlabeled target domain for video semantic segmentation. To solve the challenge, we propose a novel DA-STC method for domain adaptive video semantic segmentation, which incorporates a bidirectional multi-level spatio-temporal fusion module and a category-aware spatio-temporal feature alignment module to facilitate consistent learning for domain-invariant features. Firstly, we perform bidirectional spatio-temporal fusion at the image sequence level and shallow feature level, leading to the construction of two fused intermediate video domains. This prompts the video semantic segmentation model to consistently learn spatio-temporal features of shared patch sequences which are influenced by domain-specific contexts, thereby mitigating the feature gap between the source and target domain. Secondly, we propose a category-aware feature alignment module to promote the consistency of spatio-temporal features, facilitating adaptation to the target domain. Specifically, we adaptively aggregate the domain-specific deep features of each category along spatio-temporal dimensions, which are further constrained to achieve cross-domain intra-class feature alignment and inter-class feature separation. Extensive experiments demonstrate the effectiveness of our method, which achieves state-of-the-art mIOUs on multiple challenging benchmarks. Furthermore, we extend the proposed DA-STC to the image domain, where it also exhibits superior performance for domain adaptive semantic segmentation. The source code and models will be made available at \url{https://github.com/ZHE-SAPI/DA-STC}.

Related papers

• Exploiting Aggregation and Segregation of Representations for Domain Adaptive Human Pose Estimation [50.31351006532924]
  Human pose estimation (HPE) has received increasing attention recently due to its wide application in motion analysis, virtual reality, healthcare, etc. It suffers from the lack of labeled diverse real-world datasets due to the time- and labor-intensive annotation. We introduce a novel framework that capitalizes on both representation aggregation and segregation for domain adaptive human pose estimation.
  arXiv  Detail & Related papers  (2024-12-29T17:59:45Z)
• Adversarially Domain-adaptive Latent Diffusion for Unsupervised Semantic Segmentation [7.099012213719071]
  This work introduces a semantic segmentation method based on latent diffusion models, termed Inter-Coder Connected Latent Diffusion (ICCLD) ICCLD outperforms state-of-the-art UDA methods, achieving mIoU scores of 74.4 (GTA5$rightarrow$Cityscapes) and 67.2 ( Synthia$rightarrow$Cityscapes)
  arXiv  Detail & Related papers  (2024-12-22T04:55:41Z)
• Improving Intrusion Detection with Domain-Invariant Representation Learning in Latent Space [4.871119861180455]
  We introduce a two-phase representation learning technique using multi-task learning. We disentangle the latent space by minimizing the mutual information between the prior and latent space. We assess the model's efficacy across multiple cybersecurity datasets.
  arXiv  Detail & Related papers  (2023-12-28T17:24:13Z)
• Towards Full-scene Domain Generalization in Multi-agent Collaborative Bird's Eye View Segmentation for Connected and Autonomous Driving [49.03947018718156]
  We propose a unified domain generalization framework to be utilized during the training and inference stages of collaborative perception. We also introduce an intra-system domain alignment mechanism to reduce or potentially eliminate the domain discrepancy among connected and autonomous vehicles.
  arXiv  Detail & Related papers  (2023-11-28T12:52:49Z)
• Towards Domain-Specific Features Disentanglement for Domain Generalization [23.13095840134744]
  We propose a novel contrastive-based disentanglement method CDDG to exploit the over-looked domain-specific features. Specifically, CDDG learns to decouple inherent mutually exclusive features by leveraging them in the latent space. Experiments conducted on various benchmark datasets demonstrate the superiority of our method compared to other state-of-the-art approaches.
  arXiv  Detail & Related papers  (2023-10-04T17:51:02Z)
• Improving Anomaly Segmentation with Multi-Granularity Cross-Domain Alignment [17.086123737443714]
  Anomaly segmentation plays a pivotal role in identifying atypical objects in images, crucial for hazard detection in autonomous driving systems. While existing methods demonstrate noteworthy results on synthetic data, they often fail to consider the disparity between synthetic and real-world data domains. We introduce the Multi-Granularity Cross-Domain Alignment framework, tailored to harmonize features across domains at both the scene and individual sample levels.
  arXiv  Detail & Related papers  (2023-08-16T22:54:49Z)
• CAusal and collaborative proxy-tasKs lEarning for Semi-Supervised Domain Adaptation [20.589323508870592]
  Semi-supervised domain adaptation (SSDA) adapts a learner to a new domain by effectively utilizing source domain data and a few labeled target samples. We show that the proposed model significantly outperforms SOTA methods in terms of effectiveness and generalisability on SSDA datasets.
  arXiv  Detail & Related papers  (2023-03-30T16:48:28Z)
• A Comprehensive Survey on Source-free Domain Adaptation [69.17622123344327]
  The research of Source-Free Domain Adaptation (SFDA) has drawn growing attention in recent years. We provide a comprehensive survey of recent advances in SFDA and organize them into a unified categorization scheme. We compare the results of more than 30 representative SFDA methods on three popular classification benchmarks.
  arXiv  Detail & Related papers  (2023-02-23T06:32:09Z)
• AADG: Automatic Augmentation for Domain Generalization on Retinal Image Segmentation [1.0452185327816181]
  We propose a data manipulation based domain generalization method, called Automated Augmentation for Domain Generalization (AADG) Our AADG framework can effectively sample data augmentation policies that generate novel domains. Our proposed AADG exhibits state-of-the-art generalization performance and outperforms existing approaches.
  arXiv  Detail & Related papers  (2022-07-27T02:26:01Z)
• Amplitude Spectrum Transformation for Open Compound Domain Adaptive Semantic Segmentation [62.68759523116924]
  Open compound domain adaptation (OCDA) has emerged as a practical adaptation setting. We propose a novel feature space Amplitude Spectrum Transformation (AST)
  arXiv  Detail & Related papers  (2022-02-09T05:40:34Z)
• A New Bidirectional Unsupervised Domain Adaptation Segmentation Framework [27.13101555533594]
  unsupervised domain adaptation (UDA) techniques are proposed to bridge the gap between different domains. In this paper, we propose a bidirectional UDA framework based on disentangled representation learning for equally competent two-way UDA performances.
  arXiv  Detail & Related papers  (2021-08-18T05:25:11Z)
• AFAN: Augmented Feature Alignment Network for Cross-Domain Object Detection [90.18752912204778]
  Unsupervised domain adaptation for object detection is a challenging problem with many real-world applications. We propose a novel augmented feature alignment network (AFAN) which integrates intermediate domain image generation and domain-adversarial training. Our approach significantly outperforms the state-of-the-art methods on standard benchmarks for both similar and dissimilar domain adaptations.
  arXiv  Detail & Related papers  (2021-06-10T05:01:20Z)
• Domain Conditioned Adaptation Network [90.63261870610211]
  We propose a Domain Conditioned Adaptation Network (DCAN) to excite distinct convolutional channels with a domain conditioned channel attention mechanism. This is the first work to explore the domain-wise convolutional channel activation for deep DA networks.
  arXiv  Detail & Related papers  (2020-05-14T04:23:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.