DA-STC: Domain Adaptive Video Semantic Segmentation via Spatio-Temporal
Consistency
- URL: http://arxiv.org/abs/2311.13254v1
- Date: Wed, 22 Nov 2023 09:18:49 GMT
- Title: DA-STC: Domain Adaptive Video Semantic Segmentation via Spatio-Temporal
Consistency
- Authors: Zhe Zhang, Gaochang Wu, Jing Zhang, Chunhua Shen, Dacheng Tao, Tianyou
Chai
- Abstract summary: We propose a DA-STC method for domain adaptive video semantic segmentation, which incorporates a multi-level-level fusion module.
We also propose a category-aware feature alignment module to promote the consistency of significant-temporal features.
Our method exhibits superior performance for domain adaptive semantic segmentation.
- Score: 111.48756648371256
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Video semantic segmentation is a pivotal aspect of video representation
learning. However, significant domain shifts present a challenge in effectively
learning invariant spatio-temporal features across the labeled source domain
and unlabeled target domain for video semantic segmentation. To solve the
challenge, we propose a novel DA-STC method for domain adaptive video semantic
segmentation, which incorporates a bidirectional multi-level spatio-temporal
fusion module and a category-aware spatio-temporal feature alignment module to
facilitate consistent learning for domain-invariant features. Firstly, we
perform bidirectional spatio-temporal fusion at the image sequence level and
shallow feature level, leading to the construction of two fused intermediate
video domains. This prompts the video semantic segmentation model to
consistently learn spatio-temporal features of shared patch sequences which are
influenced by domain-specific contexts, thereby mitigating the feature gap
between the source and target domain. Secondly, we propose a category-aware
feature alignment module to promote the consistency of spatio-temporal
features, facilitating adaptation to the target domain. Specifically, we
adaptively aggregate the domain-specific deep features of each category along
spatio-temporal dimensions, which are further constrained to achieve
cross-domain intra-class feature alignment and inter-class feature separation.
Extensive experiments demonstrate the effectiveness of our method, which
achieves state-of-the-art mIOUs on multiple challenging benchmarks.
Furthermore, we extend the proposed DA-STC to the image domain, where it also
exhibits superior performance for domain adaptive semantic segmentation. The
source code and models will be made available at
\url{https://github.com/ZHE-SAPI/DA-STC}.
Related papers
- Birds of A Feather Flock Together: Category-Divergence Guidance for
Domain Adaptive Segmentation [35.63920597305474]
Unsupervised domain adaptation (UDA) aims to enhance the generalization capability of a certain model from a source domain to a target domain.
In this work, we propose an Inter-class Separation and Intra-class Aggregation (ISIA) mechanism.
By measuring the align complexity of each category, we design an Adaptive-weighted Instance Matching (AIM) strategy to further optimize the instance-level adaptation.
arXiv Detail & Related papers (2022-04-05T11:17:19Z) - Amplitude Spectrum Transformation for Open Compound Domain Adaptive
Semantic Segmentation [62.68759523116924]
Open compound domain adaptation (OCDA) has emerged as a practical adaptation setting.
We propose a novel feature space Amplitude Spectrum Transformation (AST)
arXiv Detail & Related papers (2022-02-09T05:40:34Z) - AFAN: Augmented Feature Alignment Network for Cross-Domain Object
Detection [90.18752912204778]
Unsupervised domain adaptation for object detection is a challenging problem with many real-world applications.
We propose a novel augmented feature alignment network (AFAN) which integrates intermediate domain image generation and domain-adversarial training.
Our approach significantly outperforms the state-of-the-art methods on standard benchmarks for both similar and dissimilar domain adaptations.
arXiv Detail & Related papers (2021-06-10T05:01:20Z) - More Separable and Easier to Segment: A Cluster Alignment Method for
Cross-Domain Semantic Segmentation [41.81843755299211]
We propose a new UDA semantic segmentation approach based on domain assumption closeness to alleviate the above problems.
Specifically, a prototype clustering strategy is applied to cluster pixels with the same semantic, which will better maintain associations among target domain pixels.
Experiments conducted on GTA5 and SYNTHIA proved the effectiveness of our method.
arXiv Detail & Related papers (2021-05-07T10:24:18Z) - Cluster, Split, Fuse, and Update: Meta-Learning for Open Compound Domain
Adaptive Semantic Segmentation [102.42638795864178]
We propose a principled meta-learning based approach to OCDA for semantic segmentation.
We cluster target domain into multiple sub-target domains by image styles, extracted in an unsupervised manner.
A meta-learner is thereafter deployed to learn to fuse sub-target domain-specific predictions, conditioned upon the style code.
We learn to online update the model by model-agnostic meta-learning (MAML) algorithm, thus to further improve generalization.
arXiv Detail & Related papers (2020-12-15T13:21:54Z) - Cross-Domain Grouping and Alignment for Domain Adaptive Semantic
Segmentation [74.3349233035632]
Existing techniques to adapt semantic segmentation networks across the source and target domains within deep convolutional neural networks (CNNs) do not consider an inter-class variation within the target domain itself or estimated category.
We introduce a learnable clustering module, and a novel domain adaptation framework called cross-domain grouping and alignment.
Our method consistently boosts the adaptation performance in semantic segmentation, outperforming the state-of-the-arts on various domain adaptation settings.
arXiv Detail & Related papers (2020-12-15T11:36:21Z) - Affinity Space Adaptation for Semantic Segmentation Across Domains [57.31113934195595]
In this paper, we address the problem of unsupervised domain adaptation (UDA) in semantic segmentation.
Motivated by the fact that source and target domain have invariant semantic structures, we propose to exploit such invariance across domains.
We develop two affinity space adaptation strategies: affinity space cleaning and adversarial affinity space alignment.
arXiv Detail & Related papers (2020-09-26T10:28:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.