GAAT: Geometry-Aware Alignment Transformer for Multimodal UAV Perception

• URL: http://arxiv.org/abs/2608.27971v1
• Date: Fri, 28 Aug 2026 06:36:17 GMT
• Title: GAAT: Geometry-Aware Alignment Transformer for Multimodal UAV Perception
• Abstract summary: Unmanned aerial vehicle (UAV) multimodal perception integrates visible (RGB), infrared (IR), synthetic aperture radar (SAR) and depth sensors for scene understanding under diverse conditions.<n>We propose GAAT (Geometry-Aware Alignment Transformer), an alignment-first pretrained model that estimates local correspondence reliability before cross-modal interaction.<n>We introduce UAVMeta and StateBench, which provide four acquisition-state scores derived from platform telemetry and image statistics.
• Score: 7.90322991085067
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unmanned aerial vehicle (UAV) multimodal perception integrates visible (RGB), infrared (IR), synthetic aperture radar (SAR), and depth sensors for scene understanding under diverse conditions. However, differences in optics, resolution, and mounting often limit practical systems to global or image-center alignment. After tokenization, parallax, platform motion, and lens distortion can shift corresponding patch centers across modalities, weakening the spatial correspondence assumed by dense contrastive learning and cross-modal fusion. We propose GAAT (Geometry-Aware Alignment Transformer), an alignment-first pretrained model that estimates local correspondence reliability before cross-modal interaction. GAAT introduces syncPATC, which learns patch-center consistency under synchronized view transformations without correspondence annotations. It emits geometric priors, including token and query confidence, query centers, and sub-token offsets, that identify reliable local anchors across residual misalignment. Guided by these priors, MG-Sparse-MMA performs query-mediated sparse fusion over top-K_s reliable regions, replacing dense all-patch interaction with geometry-calibrated local updates. RA-QCGCL aligns pretraining supervision with this sparse query bottleneck through reliable patch-to-patch, patch-to-query, and query-to-query contrastive branches. We introduce UAVMeta and StateBench, which provide four acquisition-state scores derived from platform telemetry and image statistics: camera reliability, observation scale, viewpoint stability, and flight maneuver complexity. Extensive experiments across six downstream tasks demonstrate consistently superior transfer performance, establishing GAAT as a state-of-the-art multimodal foundation model for UAV perception. StateBench further enables a systematic diagnosis of real-world acquisition conditions.

Related papers

• Bridging Severe Cross-Modal Misalignment: End-to-End Visible-Infrared Object Detection via Explicit Feature-Domain Affine Registration [17.74037850404483]
  We propose an end-to-end visible-infrared oriented object detector tailored for severely cross-modal geometric discrepancies.<n>The proposed JFRDet achieves 69.7% $mathrmmAP_50$ on DroneVehicle Misaligned (DVMA), which represents state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2026-08-11T09:02:37Z)
• FLORO: A Multimodal Geospatial Foundation Model for Ecological Remote Sensing Across Sensors and Scales [65.4821703903285]
  FLORO is a multimodal foundation model designed to learn transferable representations from a small but highly diverse remote sensing corpus.<n> FLORO is pretrained using masked autoencoding on a heterogeneous combination of Sentinel-1, Sentinel-2, SkySAT imagery, elevation, and UAV-derived data.<n>We evaluated FLORO on the PANGAEA benchmark under a frozen-encoder protocol across scene classification, segmentation, and regression tasks.
  arXiv  Detail & Related papers  (2026-05-27T08:55:54Z)
• CT-VIR: Continuous-Time Visual-Inertial-Ranging Fusion for Indoor Localization with Sparse Anchors [8.090575303185078]
  We propose a continuous-time state estimation method for visual-inertial-ranging (VIR) fusion localization.<n>Introductor, visual, and ranging constraints are formulated as factors in a sliding-window graph.<n> Evaluations on public datasets and real-world experiments demonstrate the effectiveness and practical potential of the proposed approach.
  arXiv  Detail & Related papers  (2026-04-16T02:23:57Z)
• Unifying UAV Cross-View Geo-Localization via 3D Geometric Perception [51.687842983240564]
  Cross-view geo-localization for Unmanned Aerial Vehicles (UAVs) remains challenging due to the severe geometric discrepancy between oblique UAV imagery and satellite maps.<n>We propose a geometry-aware UAV geo-localization framework that explicitly models the 3D scene geometry to unify coarse place recognition and fine-grained pose estimation.<n>Our method significantly outperforms state-of-the-art baselines, achieving robust meter-level localization accuracy and improved generalization in complex urban environments.
  arXiv  Detail & Related papers  (2026-04-02T08:08:41Z)
• VFM-Loc: Zero-Shot Cross-View Geo-Localization via Aligning Discriminative Visual Hierarchies [5.868908918033978]
  Cross-View Geo-Localization (CVGL) in remote sensing aims to locate a drone-view query by matching it to geo-tagged satellite images.<n>We present VFM-Loc, a training-free framework for zero-shot CVGL that leverages the generalizable visual representations from vision foundational models (VFMs)<n>Experiments demonstrate that VFM-Loc exhibits strong zero-shot accuracy on standard benchmarks and surpasses supervised methods by over 20% in Recall@1 on the challenging LO-UCV dataset with large oblique angles.
  arXiv  Detail & Related papers  (2026-03-14T09:23:59Z)
• IoUCert: Robustness Verification for Anchor-based Object Detectors [58.35703549470485]
  We introduce IoUCert, a novel formal verification framework designed specifically to overcome these bottlenecks in anchor-based object detection architectures.<n>We show that our method enables the robustness verification of realistic, anchor-based models including SSD, YOLOv2, and YOLOv3 variants against various input perturbations.
  arXiv  Detail & Related papers  (2026-03-03T14:36:46Z)
• Distributed and Consistent Multi-Robot Visual-Inertial-Ranging Odometry on Lie Groups [1.2247984232203188]
  This paper proposes a distributed visual-inertial-ranging odometry (DC-VIRO) framework that tightly fuses VIO and UWB measurements across multiple robots.<n>By leveraging a right-invariant error formulation on Lie groups, the proposed approach preserves the observability properties of standard VIO.
  arXiv  Detail & Related papers  (2026-02-22T13:07:50Z)
• Rethinking Multi-Condition DiTs: Eliminating Redundant Attention via Position-Alignment and Keyword-Scoping [61.459927600301654]
  Multi-condition control is bottlenecked by the conventional concatenate-and-attend'' strategy.<n>Our analysis reveals that much of this cross-modal interaction is spatially or semantically redundant.<n>We propose Position-aligned and Keyword-scoped Attention (PKA), a highly efficient framework designed to eliminate these redundancies.
  arXiv  Detail & Related papers  (2026-02-06T16:39:10Z)
• GSAlign: Geometric and Semantic Alignment Network for Aerial-Ground Person Re-Identification [32.31970656501684]
  Aerial-Ground person re-identification (AG-ReID) is an emerging yet challenging task that aims to match pedestrian images from drastically different viewpoints.<n>The task poses significant challenges due to extreme viewpoint discrepancies, warps, and domain gaps between aerial and ground imagery.
  arXiv  Detail & Related papers  (2025-10-25T12:16:10Z)
• Loc$^2$: Interpretable Cross-View Localization via Depth-Lifted Local Feature Matching [80.57282092735991]
  We propose an accurate and interpretable fine-grained cross-view localization method.<n>It estimates the 3 Degrees of Freedom (DoF) pose of a ground-level image by matching its local features with a reference aerial image.<n> Experiments show state-of-the-art accuracy in challenging scenarios such as cross-area testing and unknown orientation.
  arXiv  Detail & Related papers  (2025-09-11T18:52:16Z)
• PAID: Pairwise Angular-Invariant Decomposition for Continual Test-Time Adaptation [70.98107766265636]
  This paper takes the geometric attributes of pre-trained weights as a starting point, systematically analyzing three key components: magnitude, absolute angle, and pairwise angular structure.<n>We find that the pairwise angular structure remains stable across diverse corrupted domains and encodes domain-invariant semantic information, suggesting it should be preserved during adaptation.
  arXiv  Detail & Related papers  (2025-06-03T05:18:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.