Not All Patches are Equal: Sampling Matters for Visible-Infrared Pre-Training

• URL: http://arxiv.org/abs/2607.20238v1
• Date: Wed, 22 Jul 2026 14:59:02 GMT
• Title: Not All Patches are Equal: Sampling Matters for Visible-Infrared Pre-Training
• Abstract summary: Importance-Aware Sampling adjusts training emphasis based on patch reliability.<n>Experiments on multiple VIS-IR benchmarks demonstrate consistent improvements over strong baselines.
• Score: 44.905875663301515
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visible-infrared (VIS-IR) alignment is a key pre-training task for robust multi-sensor perception. Most existing methods use uniform patch-wise contrastive learning, but this can be unreliable in VIS-IR data because imaging-physics differences make some spatially paired regions inherently less comparable, and aligning them with equal strength hinders representation learning and downstream transfer. In this paper, we revisit VIS-IR pre-training from a sampling perspective and propose Importance-Aware Sampling (IAS), which adjusts training emphasis based on patch reliability. Specifically, IAS (i) derives patch weights from infrared structural cues and uses them to reweight the contrastive objective; (ii) learns a soft importance mask with a lightweight sampler, optionally warm-started from the hand-crafted prior; and (iii) employs a patch curriculum learning strategy that gradually expands from high-reliability regions to harder patches. It is worth noting that IAS is plug-and-play and works with both patch-/correlation-level alignment (e.g., UNIV-style) and image-level contrastive baselines (e.g., ImageBind-style). Extensive experiments on multiple VIS-IR benchmarks demonstrate consistent improvements over strong baselines, including for IR semantic segmentation, IR object detection and VIS semantic segmentation and cross-modal retrieval task. Code will be released on https://github.com/KlayMa527/IAS.

Related papers

• SC-Diff: Semantically Calibrated Diffusion for Visible-to-Infrared Image Translation [52.62632888060867]
  Visible-to-infrared image translation provides a practical way to expand infrared training data using abundant visible images.<n>Existing diffusion-based methods typically use semantic priors only as external conditions.<n>We propose SC-Diff, a semantically latent diffusion framework that uses semantic priors for both conditional guidance and internal self-attention calibration.
  arXiv  Detail & Related papers  (2026-08-09T07:57:40Z)
• Robust Scene Transfer for PointGoal Navigation via Privileged Sensor Guided Contrastive Learning [45.2217589644315]
  We propose a sensor-guided adaptive contrastive learning framework for visual representation learning in PointGoal navigation.<n>During training, privileged LiDAR sensing guides the contrastive objective through a geometry-aware similarity metric and adaptive temperature scaling.<n>The resulting encoder is pretrained independently, frozen, and used as the backbone for reinforcement learning.
  arXiv  Detail & Related papers  (2026-06-03T23:15:50Z)
• Exposing Vulnerabilities in Visible-Infrared VLMs: A Unified Geometric Adversarial Framework with Cross-Task Transferability [11.461885964494199]
  Vision-language models (VLMs) have achieved strong performance across diverse multimodal tasks, but their adversarial robustness in visible-infrared (VIS-IR) scenarios remains underexplored.<n>We propose CFGPatch, a curved-edge fractal geometric adversarial patch framework for attacking VIS-IR VLMs.
  arXiv  Detail & Related papers  (2026-05-21T10:15:58Z)
• Multi-Sensor Matching with HyperNetworks [14.911092205861822]
  We leverage hypernetworks to improve multimodal patch matching.<n>We introduce a lightweight descriptor-learning architecture that augments a Siamese CNN.<n>We also release GAP-VIR, a cross-platform (ground/aerial) VIS-IR patch dataset with 500K pairs.
  arXiv  Detail & Related papers  (2026-01-18T09:19:33Z)
• PANICL: Mitigating Over-Reliance on Single Prompt in Visual In-Context Learning [38.70994263844236]
  Visual In-Context Learning (VICL) uses input-output image pairs, referred to as in-context pairs (or examples), as prompts alongside query images to guide models in performing diverse vision tasks.<n>VICL often suffers from over-reliance on a single in-context pair, which can lead to biased and unstable predictions.<n>We introduce PAtch-based $k$-Nearest neighbor visual In-Context Learning (PANICL), a general training-free framework that mitigates this issue by leveraging multiple in-context pairs.
  arXiv  Detail & Related papers  (2025-09-26T06:13:40Z)
• FreRA: A Frequency-Refined Augmentation for Contrastive Learning on Time Series Classification [56.925103708982164]
  We present a novel perspective from the frequency domain and identify three advantages for downstream classification: global, independent, and compact.<n>We propose the lightweight yet effective Frequency Refined Augmentation (FreRA) tailored for time series contrastive learning on classification tasks.<n>FreRA consistently outperforms ten leading baselines on time series classification, anomaly detection, and transfer learning tasks.
  arXiv  Detail & Related papers  (2025-05-29T07:18:28Z)
• Multi-Domain Biometric Recognition using Body Embeddings [51.36007967653781]
  We show that body embeddings perform better than face embeddings in medium-wave infrared (MWIR) and long-wave infrared (LWIR) domains.<n>We leverage a vision transformer architecture to establish benchmark results on the IJB-MDF dataset.<n>We also show that finetuning a body model, pretrained exclusively on VIS data, with a simple combination of cross-entropy and triplet losses achieves state-of-the-art mAP scores.
  arXiv  Detail & Related papers  (2025-03-13T22:38:18Z)
• IV-tuning: Parameter-Efficient Transfer Learning for Infrared-Visible Tasks [57.940401122426664]
  Under the full fine-tuning paradigm, the feature space becomes highly constrained and low-ranked, which has been proven to seriously impair generalization.<n>We propose IV-tuning, to parameter-efficiently harness PVMs for various IR-VIS downstream tasks, including salient object detection, semantic segmentation, and object detection.<n>Compared with the full fine-tuning baselines and existing IR-VIS methods, IV-tuning facilitates the learning of complementary information between infrared and visible modalities with less than 3% of the backbone parameters.
  arXiv  Detail & Related papers  (2024-12-21T14:54:41Z)
• Exploring Text-Guided Single Image Editing for Remote Sensing Images [30.66938568608091]
  This paper proposes a text-guided RSI editing method and can be trained using only a single image.<n>A multi-scale training approach is adopted to preserve consistency without the need for training on extensive benchmarks.<n>The proposed method offers significant advantages in both CLIP scores and subjective evaluations compared to existing methods.
  arXiv  Detail & Related papers  (2024-05-09T13:45:04Z)
• Self-Promoted Supervision for Few-Shot Transformer [178.52948452353834]
  Self-promoted sUpervisioN (SUN) is a few-shot learning framework for vision transformers (ViTs) SUN pretrains the ViT on the few-shot learning dataset and then uses it to generate individual location-specific supervision for guiding each patch token. Experiments show that SUN using ViTs significantly surpasses other few-shot learning frameworks with ViTs and is the first one that achieves higher performance than those CNN state-of-the-arts.
  arXiv  Detail & Related papers  (2022-03-14T12:53:27Z)
• Dense Label Encoding for Boundary Discontinuity Free Rotation Detection [69.75559390700887]
  This paper explores a relatively less-studied methodology based on classification. We propose new techniques to push its frontier in two aspects. Experiments and visual analysis on large-scale public datasets for aerial images show the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-11-19T05:42:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.