Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos

• URL: http://arxiv.org/abs/2607.19857v1
• Date: Wed, 22 Jul 2026 07:45:50 GMT
• Title: Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos
• Abstract summary: Language-guided aerial perception aims to understand user-specified tiny targets in complex unmanned aerial vehicle (UAV) scenes.<n>In real UAV deployment, the UAV must respond while it flies, so such perception runs in an online streaming manner, where frames arrive sequentially and the model responds to each one without access to future frames.<n>We address the tiny object and streaming challenges from both data and method perspectives.
• Score: 62.42252327381627
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language-guided aerial perception aims to understand user-specified tiny targets in complex unmanned aerial vehicle (UAV) scenes. In real UAV deployment, the UAV must respond while it flies, so such perception runs in an online streaming manner, where frames arrive sequentially and the model responds to each one without access to future frames. However, applying current Multimodal Large Language Models (MLLMs) to this setting raises two challenges. First, targets viewed from the air are often tiny, yet the visual compression in existing MLLMs treats all regions equally and discards their fine-grained details. Second, understanding a continuous stream requires past-frame context, yet retaining the entire history is infeasible on resource-constrained onboard hardware, whereas discarding it causes the target to drift or disappear. We address the tiny object and streaming challenges from both data and method perspectives. From the data perspective, we present \textbf{DroneEyes}, the \textbf{first} pixel-level and open-vocabulary referring-segmentation dataset for tiny aerial targets, comprising $2,140$ high-definition videos and $176,623$ pairs across Object Description and Referring Expression tasks, with dense per-frame masks. From the method perspective, we propose \textbf{SkyAnchor}, an MLLM with two designs to the above challenges: a Semantics-Aware Token Router that preserves small-target under a reduced visual-token budget, and a Hierarchical Memory Bank that keeps the target consistently understood on streams.

Related papers

• SkyVLaM: Multimodal Large Language Model for UAV Video Understanding in Remote Sensing [10.503838482038736]
  We propose SkyVLaM, a multimodal large language model for UAV video understanding.<n>SkyVLaM constructs sparse tokens directly from patch-level video representations through a temporal basis perceiver.<n>We further build SkyVid, consisting of SkyVid-VGCG and SkyVid-RVOS for video grounded conversation generation and referring video object segmentation.
  arXiv  Detail & Related papers  (2026-07-19T19:07:19Z)
• City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning [18.827215649935468]
  We build the first multidomain perception outdoor scene understanding dataset, named textbfunderlineSVM-City.<n>It contains $420$k images and $4, 811$M point clouds with $567$k question-answering pairs from vehicles, low-altitude drones, high-altitude aerial planes, and satellite.<n> Experimental results show City-VLM achieves $18.14 %$ performance surpassing existing LVLMs in question-answering tasks averagely.
  arXiv  Detail & Related papers  (2025-07-17T05:21:21Z)
• VideoMolmo: Spatio-Temporal Grounding Meets Pointing [66.19964563104385]
  VideoMolmo is a model tailored for fine-grained pointing of video sequences.<n>A novel temporal mask fusion employs SAM2 for bidirectional point propagation.<n>To evaluate the generalization of VideoMolmo, we introduce VPoMolS-temporal, a challenging out-of-distribution benchmark spanning five real-world scenarios.
  arXiv  Detail & Related papers  (2025-06-05T17:59:29Z)
• AeroReformer: Aerial Referring Transformer for UAV-based Referring Image Segmentation [9.55871636831991]
  We propose a novel framework for UAV referring image segmentation (UAV-RIS)<n>AeroReformer features a Vision-Language Cross-Attention Module (VLCAM) for effective cross-modal understanding and a Rotation-Aware Multi-Scale Fusion decoder.<n>Experiments on two newly developed datasets demonstrate the superiority of AeroReformer over existing methods.
  arXiv  Detail & Related papers  (2025-02-23T18:49:00Z)
• VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks [89.24440488456405]
  VisionLLM v2 is an end-to-end generalist multimodal large model (MLLM)<n>It unifies visual perception, understanding, and generation within a single framework.
  arXiv  Detail & Related papers  (2024-06-12T16:44:50Z)
• Elysium: Exploring Object-level Perception in Videos via MLLM [11.02937968639935]
  We propose an end-to-end trainable MLLM that attempts to conduct object-level tasks in videos without requiring any additional plug-in or expert models. Elysium: Exploring Object-level Perception in Videos via MLLM is an end-to-end trainable MLLM that attempts to conduct object-level tasks in videos without requiring any additional plug-in or expert models.
  arXiv  Detail & Related papers  (2024-03-25T09:17:15Z)
• Redundancy-aware Transformer for Video Question Answering [71.98116071679065]
  We propose a novel transformer-based architecture, that aims to model VideoQA in a redundancy-aware manner. To address the neighboring-frame redundancy, we introduce a video encoder structure that emphasizes the object-level change in neighboring frames. As for the cross-modal redundancy, we equip our fusion module with a novel adaptive sampling, which explicitly differentiates the vision-language interactions.
  arXiv  Detail & Related papers  (2023-08-07T03:16:24Z)
• Full-Duplex Strategy for Video Object Segmentation [141.43983376262815]
  Full- Strategy Network (FSNet) is a novel framework for video object segmentation (VOS) Our FSNet performs the crossmodal feature-passing (i.e., transmission and receiving) simultaneously before fusion decoding stage. We show that our FSNet outperforms other state-of-the-arts for both the VOS and video salient object detection tasks.
  arXiv  Detail & Related papers  (2021-08-06T14:50:50Z)
• Rethinking Cross-modal Interaction from a Top-down Perspective for Referring Video Object Segmentation [140.4291169276062]
  Referring video object segmentation (RVOS) aims to segment video objects with the guidance of natural language reference. Previous methods typically tackle RVOS through directly grounding linguistic reference over the image lattice. In this work, we put forward a two-stage, top-down RVOS solution. First, an exhaustive set of object tracklets is constructed by propagating object masks detected from several sampled frames to the entire video. Second, a Transformer-based tracklet-language grounding module is proposed, which models instance-level visual relations and cross-modal interactions simultaneously and efficiently.
  arXiv  Detail & Related papers  (2021-06-02T10:26:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.