Unlocking Multi-Spectral Data for Multi-Modal Models with Guided Inputs and Chain-of-Thought Reasoning

• URL: http://arxiv.org/abs/2604.21032v1
• Date: Wed, 22 Apr 2026 19:23:52 GMT
• Title: Unlocking Multi-Spectral Data for Multi-Modal Models with Guided Inputs and Chain-of-Thought Reasoning
• Abstract summary: Multi-spectral imagery is a valuable input signal for Remote Sensing applications.<n>Large Multi-modal Models (LMMs) are typically trained on RGB images, limiting their applicability to the RGB domain.<n>We propose a novel training-free approach that introduces multi-spectral data within the inference pipeline of standard RGB-only LMMs.
• Score: 21.749879619777687
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-spectral imagery is a valuable input signal for Remote Sensing applications, such as land-use and land-cover classification and environmental monitoring. However, generalist Large Multi-modal Models (LMMs) are typically trained on RGB images, limiting their applicability to the RGB domain. At the same time, training multi-spectral multi-modal models is expensive and produces uniquely specialized models. To address this, we propose a novel training-free approach that introduces multi-spectral data within the inference pipeline of standard RGB-only LMMs, allowing large gains in performance. Our approach leverages the LMMs' understanding of the visual space by adapting non-RGB inputs to that space and injecting domain-specific information and Chain-of-Thought reasoning as instructions. We demonstrate this with the Gemini 2.5 model and observe strong Zero-Shot performance gains on popular Remote Sensing benchmarks. These results highlight the potential for geospatial professionals to leverage powerful generalist models for specialized sensor inputs, benefiting from rich reasoning capabilities grounded in specialized data.

Related papers

• SpectraDINO: Bridging the Spectral Gap in Vision Foundation Models via Lightweight Adapters [1.2622634782102324]
  Vision Foundation Models (VFMs) pretrained on large-scale RGB data have demonstrated remarkable representation quality.<n>Their applicability to multispectral imaging spanning Near-Infrared (NIR), Short-Wave Infrared (SWIR), and Long-Wave Infrared (LWIR) remains largely unexplored.<n>We present SpectraDINO, a multispectral VFM that bridges this spectral gap by extending DINOv2 ViT backbones to beyond-visible modalities.
  arXiv  Detail & Related papers  (2026-05-04T06:09:13Z)
• MMOT: The First Challenging Benchmark for Drone-based Multispectral Multi-Object Tracking [30.3437683353074]
  MMOT is the first benchmark for drone-based multispectral multi-object tracking.<n>It features 125 video sequences with over 488.8K annotations across eight categories.<n>To better extract spectral features and leverage oriented annotations, we present a multispectral and orientation-aware MOT scheme.
  arXiv  Detail & Related papers  (2025-10-14T14:25:17Z)
• Zero-Shot Multi-Spectral Learning: Reimagining a Generalist Multimodal Gemini 2.5 Model for Remote Sensing Applications [18.162960869640646]
  We propose a training-free approach which introduces new multi-spectral data in a Zero-Shot-only mode.<n>We observe strong Zero-Shot performance gains of the approach on popular Remote Sensing benchmarks for land cover and land use classification.
  arXiv  Detail & Related papers  (2025-09-23T14:40:52Z)
• HyPSAM: Hybrid Prompt-driven Segment Anything Model for RGB-Thermal Salient Object Detection [75.406055413928]
  We propose a novel prompt-driven segment anything model (HyPSAM) for RGB-T SOD.<n> DFNet employs dynamic convolution and multi-branch decoding to facilitate adaptive cross-modality interaction.<n>Experiments on three public datasets demonstrate that our method achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-09-23T07:32:11Z)
• Vision-G1: Towards General Vision Language Reasoning with Multi-Domain Data Curation [64.23194519770897]
  We build a comprehensive RL-ready visual reasoning dataset from 46 data sources across 8 dimensions.<n>We propose an influence function based data selection and difficulty based filtering strategy to identify high-quality training samples from this dataset.<n>We train the VLM, referred to as Vision-G1, using multi-round RL with a data curriculum to iteratively improve its visual reasoning capabilities.
  arXiv  Detail & Related papers  (2025-08-18T07:24:33Z)
• Generalizable Multispectral Land Cover Classification via Frequency-Aware Mixture of Low-Rank Token Experts [22.75047167955269]
  We introduce Land-MoE, a novel approach for multispectral land cover classification (MLCC)<n>Land-MoE comprises two key modules: the mixture of low-rank token experts (MoLTE) and frequency-aware filters (FAF)
  arXiv  Detail & Related papers  (2025-05-20T08:52:28Z)
• Emergent Visual Grounding in Large Multimodal Models Without Grounding Supervision [66.40106264913082]
  Current large multimodal models (LMMs) face challenges in grounding, which requires the model to relate language components to visual entities.<n>We find that the grounding ability can in fact emerge in LMMs trained without explicit grounding supervision.<n>We propose DIFFLMM, an LMM utilizing a diffusion-based visual encoder, as opposed to the standard CLIP visual encoder, and trained with the same weak supervision.
  arXiv  Detail & Related papers  (2024-10-10T17:59:55Z)
• When Pedestrian Detection Meets Multi-Modal Learning: Generalist Model and Benchmark Dataset [40.24765100535353]
  This paper introduces MMPedestron, a novel generalist model for multimodal perception. The proposed approach comprises a unified encoder for modal representation and fusion and a general head for pedestrian detection. With multi-modal joint training, our model achieves state-of-the-art performance on a wide range of pedestrian detection benchmarks.
  arXiv  Detail & Related papers  (2024-07-14T09:16:49Z)
• XTrack: Multimodal Training Boosts RGB-X Video Object Trackers [88.72203975896558]
  It is crucial to ensure that knowledge gained from multimodal sensing is effectively shared.<n>Similar samples across different modalities have more knowledge to share than otherwise.<n>We propose a method for RGB-X tracker during inference, with an average +3% precision improvement over the current SOTA.
  arXiv  Detail & Related papers  (2024-05-28T03:00:58Z)
• SpectralGPT: Spectral Remote Sensing Foundation Model [60.023956954916414]
  A universal RS foundation model, named SpectralGPT, is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT) Compared to existing foundation models, SpectralGPT accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS big data. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS big data applications within the field of geoscience.
  arXiv  Detail & Related papers  (2023-11-13T07:09:30Z)
• HiDAnet: RGB-D Salient Object Detection via Hierarchical Depth Awareness [2.341385717236931]
  We propose a novel Hierarchical Depth Awareness network (HiDAnet) for RGB-D saliency detection. Our motivation comes from the observation that the multi-granularity properties of geometric priors correlate well with the neural network hierarchies. Our HiDAnet performs favorably over the state-of-the-art methods by large margins.
  arXiv  Detail & Related papers  (2023-01-18T10:00:59Z)
• Dual Swin-Transformer based Mutual Interactive Network for RGB-D Salient Object Detection [67.33924278729903]
  In this work, we propose Dual Swin-Transformer based Mutual Interactive Network. We adopt Swin-Transformer as the feature extractor for both RGB and depth modality to model the long-range dependencies in visual inputs. Comprehensive experiments on five standard RGB-D SOD benchmark datasets demonstrate the superiority of the proposed DTMINet method.
  arXiv  Detail & Related papers  (2022-06-07T08:35:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.