Exploring the Common Appearance-Boundary Adaptation for Nighttime Optical Flow

• URL: http://arxiv.org/abs/2401.17642v1
• Date: Wed, 31 Jan 2024 07:51:52 GMT
• Title: Exploring the Common Appearance-Boundary Adaptation for Nighttime Optical Flow
• Authors: Hanyu Zhou, Yi Chang, Haoyue Liu, Wending Yan, Yuxing Duan, Zhiwei Shi, Luxin Yan
• Abstract summary: We propose a novel appearance-boundary adaptation framework for nighttime optical flow. In appearance adaptation, we embed the auxiliary daytime image and the nighttime image into a reflectance-aligned common space. We find that motion of the two reflectance maps are very similar, benefiting us to consistently transfer motion appearance knowledge from daytime to nighttime domain.
• Score: 17.416185015412175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We investigate a challenging task of nighttime optical flow, which suffers from weakened texture and amplified noise. These degradations weaken discriminative visual features, thus causing invalid motion feature matching. Typically, existing methods employ domain adaptation to transfer knowledge from auxiliary domain to nighttime domain in either input visual space or output motion space. However, this direct adaptation is ineffective, since there exists a large domain gap due to the intrinsic heterogeneous nature of the feature representations between auxiliary and nighttime domains. To overcome this issue, we explore a common-latent space as the intermediate bridge to reinforce the feature alignment between auxiliary and nighttime domains. In this work, we exploit two auxiliary daytime and event domains, and propose a novel common appearance-boundary adaptation framework for nighttime optical flow. In appearance adaptation, we employ the intrinsic image decomposition to embed the auxiliary daytime image and the nighttime image into a reflectance-aligned common space. We discover that motion distributions of the two reflectance maps are very similar, benefiting us to consistently transfer motion appearance knowledge from daytime to nighttime domain. In boundary adaptation, we theoretically derive the motion correlation formula between nighttime image and accumulated events within a spatiotemporal gradient-aligned common space. We figure out that the correlation of the two spatiotemporal gradient maps shares significant discrepancy, benefitting us to contrastively transfer boundary knowledge from event to nighttime domain. Moreover, appearance adaptation and boundary adaptation are complementary to each other, since they could jointly transfer global motion and local boundary knowledge to the nighttime domain.

Related papers

• Beam Spliter and Localization Induced by Controlled Perturbations after Time Boundary [2.7892599615881144]
  Recent investigation into the phenomena of refraction and reflection at temporal boundaries has attracted considerable scholarly interest. We have delineated a temporal boundary effect amenable to external control through a specifically tailored driving force. We then unveil the phenomenon of beam splitting-both in time refraction and reflection induced by perturbing the lattice's hopping parameter over time.
  arXiv  Detail & Related papers  (2025-03-20T05:43:37Z)
• Bridge Frame and Event: Common Spatiotemporal Fusion for High-Dynamic Scene Optical Flow [21.821959971338767]
  We propose a novel common modality fusion between frame and event modalities for high-dynamic scene optical flow. In motion fusion, we discover that the frame-based motion possesses spatially dense but temporally discontinuous correlation, while the event-based motion has sparse but temporally continuous correlation.
  arXiv  Detail & Related papers  (2025-03-10T07:16:32Z)
• Exploring Reliable Matching with Phase Enhancement for Night-time Semantic Segmentation [58.180226179087086]
  We propose a novel end-to-end optimized approach, named NightFormer, tailored for night-time semantic segmentation. Specifically, we design a pixel-level texture enhancement module to acquire texture-aware features hierarchically with phase enhancement and amplified attention. Our proposed method performs favorably against state-of-the-art night-time semantic segmentation methods.
  arXiv  Detail & Related papers  (2024-08-25T13:59:31Z)
• Similarity Min-Max: Zero-Shot Day-Night Domain Adaptation [52.923298434948606]
  Low-light conditions not only hamper human visual experience but also degrade the model's performance on downstream vision tasks. This paper challenges a more complicated scenario with border applicability, i.e., zero-shot day-night domain adaptation. We propose a similarity min-max paradigm that considers them under a unified framework.
  arXiv  Detail & Related papers  (2023-07-17T18:50:15Z)
• Local-Global Temporal Difference Learning for Satellite Video Super-Resolution [55.69322525367221]
  We propose to exploit the well-defined temporal difference for efficient and effective temporal compensation. To fully utilize the local and global temporal information within frames, we systematically modeled the short-term and long-term temporal discrepancies. Rigorous objective and subjective evaluations conducted across five mainstream video satellites demonstrate that our method performs favorably against state-of-the-art approaches.
  arXiv  Detail & Related papers  (2023-04-10T07:04:40Z)
• Unsupervised Hierarchical Domain Adaptation for Adverse Weather Optical Flow [18.900658568158054]
  We propose the first unsupervised framework for adverse weather optical flow via hierarchical motion-boundary adaptation. Our key insight is that adverse weather does not change the intrinsic optical flow of the scene, but causes a significant difference for the warp error between clean and degraded images.
  arXiv  Detail & Related papers  (2023-03-24T02:17:51Z)
• Spatiotemporal Multi-scale Bilateral Motion Network for Gait Recognition [3.1240043488226967]
  In this paper, motivated by optical flow, the bilateral motion-oriented features are proposed. We develop a set of multi-scale temporal representations that force the motion context to be richly described at various levels of temporal resolution.
  arXiv  Detail & Related papers  (2022-09-26T01:36:22Z)
• Cross-Domain Correlation Distillation for Unsupervised Domain Adaptation in Nighttime Semantic Segmentation [17.874336775904272]
  We propose a novel domain adaptation framework via cross-domain correlation distillation, called CCDistill. We extract the content and style knowledge contained in features, calculate the degree of inherent or illumination difference between two images. Experiments on Dark Zurich and ACDC demonstrate that CCDistill achieves the state-of-the-art performance for nighttime semantic segmentation.
  arXiv  Detail & Related papers  (2022-05-02T12:42:04Z)
• Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixing [55.73722120043086]
  We introduce Contrast and Mix (CoMix), a new contrastive learning framework that aims to learn discriminative invariant feature representations for unsupervised video domain adaptation. First, we utilize temporal contrastive learning to bridge the domain gap by maximizing the similarity between encoded representations of an unlabeled video at two different speeds. Second, we propose a novel extension to the temporal contrastive loss by using background mixing that allows additional positives per anchor, thus adapting contrastive learning to leverage action semantics shared across both domains.
  arXiv  Detail & Related papers  (2021-10-28T14:03:29Z)
• Learning Cross-modal Contrastive Features for Video Domain Adaptation [138.75196499580804]
  We propose a unified framework for video domain adaptation, which simultaneously regularizes cross-modal and cross-domain feature representations. Specifically, we treat each modality in a domain as a view and leverage the contrastive learning technique with properly designed sampling strategies.
  arXiv  Detail & Related papers  (2021-08-26T18:14:18Z)
• Exploring Rich and Efficient Spatial Temporal Interactions for Real Time Video Salient Object Detection [87.32774157186412]
  Main stream methods formulate their video saliency mainly from two independent venues, i.e., the spatial and temporal branches. In this paper, we propose atemporal network to achieve such improvement in a full interactive fashion. Our method is easy to implement yet effective, achieving high quality video saliency detection in real-time speed with 50 FPS.
  arXiv  Detail & Related papers  (2020-08-07T03:24:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.