Related papers: Latent Space Guided Scenario Sampling for Multimodal Segmentation Under Missing Modalities

Latent Space Guided Scenario Sampling for Multimodal Segmentation Under Missing Modalities

URL: http://arxiv.org/abs/2605.20372v1
Date: Tue, 19 May 2026 18:23:22 GMT
Title: Latent Space Guided Scenario Sampling for Multimodal Segmentation Under Missing Modalities
Abstract summary: In real-world remote sensing applications, one or more modalities may be unavailable due to sensor failures, adverse atmospheric conditions, or data acquisition problems.<n>We propose a novel training strategy that learns a scenario sampling distribution directly from the pretrained latent space.<n>We evaluate this strategy on three remote sensing image sets, namely DSTL, Potsdam, and Hunan, using CBC-SLP, CBC, and CMX backbones.
Score: 5.097809301149341
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Multimodal semantic segmentation benefits remote sensing analysis by combining complementary information from different sensor modalities. In real-world remote sensing applications, one or more modalities may be unavailable due to sensor failures, adverse atmospheric conditions, or data acquisition problems. Even with pretrained multimodal representations and existing fine-tuning or adaptation strategies, performance may remain limited because all modality availability scenarios are typically treated as equally informative during training. In this paper, we propose a novel training strategy that learns a scenario sampling distribution directly from the pretrained latent space. Instead of relying on uniform random modality dropout, the proposed method guides fine-tuning toward more informative modality availability scenarios. More specifically, we quantify the effect of each scenario independently based on the distortion it induces in the shared latent representation. We then capture scenario relations using a radial basis function kernel and derive refined scenario scores through a regularized kernel smoothing. These scores are then converted into a probability distribution during scenario sampling for fine-tuning. We evaluate this strategy on three remote sensing image sets, namely DSTL, Potsdam, and Hunan, using CBC-SLP, CBC, and CMX backbones. The experimental results with different image sets and backbones show that our method outperforms standard fine-tuning and LoRA-based adaptation. These findings suggest that the pretrained latent representation can serve as an effective basis for sampling during missing modality fine-tuning. Code is available at https://github.com/iremulku/Latent-Space-Guided-Scenario-Sampling

Related papers

TapSampling: Inference-Time Sampling with a Task-Progress-Understanding Verifier for Robotic Manipulation [61.35569005726248]
Existing embodied control research demonstrates remarkable performance improvements by scaling training data and model size.<n>Non-deterministic generative models, such as diffusion and autoregressive models, have been widely adopted in the field of embodied control.<n>We propose textbfTapSampling, a plug-and-play framework for inference-time sampling.
arXiv Detail & Related papers (2026-05-25T08:03:31Z)
Towards Generalizable Deepfake Detection via Real Distribution Bias Correction [62.89674593681203]
We introduce the Real Distribution Bias Correction (RDBC) framework, which consists of two key components.<n>RDBC achieves state-of-the-art performance in both in-domain and cross-domain deepfake detection.
arXiv Detail & Related papers (2026-03-14T16:11:00Z)
An LLM-driven Scenario Generation Pipeline Using an Extended Scenic DSL for Autonomous Driving Safety Validation [4.602386383455713]
Real-world crash reports are valuable for scenario-based testing of autonomous driving systems.<n>Current methods cannot effectively translate this multimodal data into precise, executable simulation scenarios.<n>We propose a scalable and verifiable pipeline that uses a large language model and a probabilistic intermediate representation.
arXiv Detail & Related papers (2026-02-24T07:44:26Z)
Exploring Semantic Clustering and Similarity Search for Heterogeneous Traffic Scenario Graph [41.2584175136191]
We first propose an expressive and flexible heterogeneous,temporal graph model for representing traffic scenarios.<n>We then propose a self-supervised method to learn a universal embedding space for scenario graphs.<n>In particular, we implement contrastive learning alongside a bootstrapping-based approach and evaluate their suitability for the scenario space.
arXiv Detail & Related papers (2025-07-07T15:10:03Z)
Towards Predicting Any Human Trajectory In Context [20.810620293371027]
TrajICL is an In-Context Learning (ICL) framework for pedestrian trajectory prediction.<n>It enables adaptation without fine-tuning on the scenario-specific data at inference time.<n>We train our model on a large-scale synthetic dataset to enhance its prediction ability.
arXiv Detail & Related papers (2025-06-01T07:18:47Z)
Downstream-Pretext Domain Knowledge Traceback for Active Learning [138.02530777915362]
We propose a downstream-pretext domain knowledge traceback (DOKT) method that traces the data interactions of downstream knowledge and pre-training guidance. DOKT consists of a traceback diversity indicator and a domain-based uncertainty estimator. Experiments conducted on ten datasets show that our model outperforms other state-of-the-art methods.
arXiv Detail & Related papers (2024-07-20T01:34:13Z)
XTrack: Multimodal Training Boosts RGB-X Video Object Trackers [88.72203975896558]
It is crucial to ensure that knowledge gained from multimodal sensing is effectively shared.<n>Similar samples across different modalities have more knowledge to share than otherwise.<n>We propose a method for RGB-X tracker during inference, with an average +3% precision improvement over the current SOTA.
arXiv Detail & Related papers (2024-05-28T03:00:58Z)
Sample and Predict Your Latent: Modality-free Sequential Disentanglement via Contrastive Estimation [2.7759072740347017]
We introduce a self-supervised sequential disentanglement framework based on contrastive estimation with no external signals. In practice, we propose a unified, efficient, and easy-to-code sampling strategy for semantically similar and dissimilar views of the data. Our method presents state-of-the-art results in comparison to existing techniques.
arXiv Detail & Related papers (2023-05-25T10:50:30Z)
Generalized Differentiable RANSAC [95.95627475224231]
$nabla$-RANSAC is a differentiable RANSAC that allows learning the entire randomized robust estimation pipeline. $nabla$-RANSAC is superior to the state-of-the-art in terms of accuracy while running at a similar speed to its less accurate alternatives.
arXiv Detail & Related papers (2022-12-26T15:13:13Z)
Simpler Certified Radius Maximization by Propagating Covariances [39.851641822878996]
We show an algorithm for maximizing the certified radius on datasets including Cifar-10, ImageNet, and Places365. We show how satisfying these criteria yields an algorithm for maximizing the certified radius on datasets with moderate depth, with a small compromise in overall accuracy.
arXiv Detail & Related papers (2021-04-13T01:38:36Z)
SMART: Simultaneous Multi-Agent Recurrent Trajectory Prediction [72.37440317774556]
We propose advances that address two key challenges in future trajectory prediction. multimodality in both training data and predictions and constant time inference regardless of number of agents.
arXiv Detail & Related papers (2020-07-26T08:17:10Z)
Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation [72.40827239394565]
We propose to compute features only at sparsely sampled locations. We then densely reconstruct the feature map with an efficient procedure. The presented network is experimentally shown to save substantial computation while maintaining accuracy over a variety of computer vision tasks.
arXiv Detail & Related papers (2020-03-19T15:36:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.