QueenVIS: Rethinking Image-Only Training for Video Instance Segmentation via Query Enrichment

• URL: http://arxiv.org/abs/2607.24598v1
• Date: Mon, 27 Jul 2026 16:00:03 GMT
• Title: QueenVIS: Rethinking Image-Only Training for Video Instance Segmentation via Query Enrichment
• Abstract summary: Video instance segmentation (VIS) requires models to detect, segment, and track object identities across frames.<n>QueenVIS introduces a query-centric framework for strengthening image-trained VIS.<n>Our findings suggest that strengthening the discriminative power and temporal stability of object queries is an important, underexplored axis for VIS.
• Score: 1.8184803929988063
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video instance segmentation (VIS) requires models to detect, segment, and track object identities across frames, and most methods enforce temporal consistency through video-level supervision. Image-only training approaches, with MinVIS as one prominent example, have challenged this assumption, reaching competitive VIS without video training by treating frames as independent images and associating instances only at inference. The field has nonetheless moved toward ever more elaborate video-trained trackers, which depend on costly identity-consistent annotations, leaving the image-only direction under-explored. A diagnostic analysis identifies object query quality as the bottleneck: queries trained only to localize objects within a frame drift apart across frames and destabilize association. QueenVIS introduces a query-centric framework for strengthening image-trained VIS. During single-frame training, we enrich Mask2Former queries with two auxiliary heads: a feature-prediction loss that aligns each query with the pooled backbone descriptor of its instance, and a center-prediction loss that injects spatial structure. Both heads are discarded at inference, adding zero parameters, and temporal identity is maintained by a training-free query-propagation and memory-bank scheme. On YouTube-VIS and OVIS with a ResNet-50 backbone, QueenVIS improves over MinVIS, up to +6.7 AP on YouTube-VIS, +4.8 AP on OVIS, and +10.3 AP on the long-sequence YouTube-VIS split. QueenVIS achieves 50.9 AP on YouTube-VIS and remains competitive with recent video-supervised state-of-the-art, without processing a single video clip during training. Our findings suggest that strengthening the discriminative power and temporal stability of object queries is an important, underexplored axis for VIS. Code and models: https://github.com/ArianKheir/QueenVIS

Related papers

• SA-VIS: Sparse frame Annotations for training Video Instance Segmentation [58.2561806729996]
  Recent online video instance segmentation (VIS) methods have achieved impressive results, thus becoming the preferred approach to segment instances in videos.<n>However, a training setup of VIS is expensive in the sense of compute as well as dense annotations required.<n>We argue that the effective modeling of the instances and their evolution in videos do not require densely annotated frames.<n>This simple low-compute module provides tremendous learning capability in using sparse video frame labels for end-to-end training.
  arXiv  Detail & Related papers  (2026-06-18T12:03:04Z)
• UVIS: Unsupervised Video Instance Segmentation [65.46196594721545]
  Videocaption instance segmentation requires classifying, segmenting, and tracking every object across video frames. We propose UVIS, a novel Unsupervised Video Instance (UVIS) framework that can perform video instance segmentation without any video annotations or dense label-based pretraining. Our framework consists of three essential steps: frame-level pseudo-label generation, transformer-based VIS model training, and query-based tracking.
  arXiv  Detail & Related papers  (2024-06-11T03:05:50Z)
• BoxVIS: Video Instance Segmentation with Box Annotations [15.082477136581153]
  We adapt the state-of-the-art pixel-supervised VIS models to a box-supervised VIS baseline and observe slight performance degradation. We propose a box-center guided spatial-temporal pairwise affinity loss to predict instance masks for better spatial and temporal consistency. It exhibits comparable instance mask prediction performance and better generalization ability than state-of-the-art pixel-supervised VIS models by using only 16% of their annotation time and cost.
  arXiv  Detail & Related papers  (2023-03-26T04:04:58Z)
• A Generalized Framework for Video Instance Segmentation [49.41441806931224]
  The handling of long videos with complex and occluded sequences has emerged as a new challenge in the video instance segmentation (VIS) community. We propose a Generalized framework for VIS, namely GenVIS, that achieves state-of-the-art performance on challenging benchmarks. We evaluate our approach on popular VIS benchmarks, achieving state-of-the-art results on YouTube-VIS 2019/2021/2022 and Occluded VIS (OVIS)
  arXiv  Detail & Related papers  (2022-11-16T11:17:19Z)
• MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training [84.81566912372328]
  MinVIS is a minimal video instance segmentation framework. It achieves state-of-the-art VIS performance with neither video-based architectures nor training procedures.
  arXiv  Detail & Related papers  (2022-08-03T17:50:42Z)
• DeVIS: Making Deformable Transformers Work for Video Instance Segmentation [4.3012765978447565]
  Video Instance (VIS) jointly tackles multi-object detection, tracking, and segmentation in video sequences. Transformers recently allowed to cast the entire VIS task as a single set-prediction problem. Deformable attention provides a more efficient alternative but its application to the temporal domain or the segmentation task have not yet been explored.
  arXiv  Detail & Related papers  (2022-07-22T14:27:45Z)
• Efficient Video Instance Segmentation via Tracklet Query and Proposal [62.897552852894854]
  Video Instance aims to simultaneously classify, segment, and track multiple object instances in videos. Most clip-level methods are neither end-to-end learnable nor real-time. This paper proposes EfficientVIS, a fully end-to-end framework with efficient training and inference.
  arXiv  Detail & Related papers  (2022-03-03T17:00:11Z)
• Crossover Learning for Fast Online Video Instance Segmentation [53.5613957875507]
  We present a novel crossover learning scheme that uses the instance feature in the current frame to pixel-wisely localize the same instance in other frames. To our knowledge, CrossVIS achieves state-of-the-art performance among all online VIS methods and shows a decent trade-off between latency and accuracy.
  arXiv  Detail & Related papers  (2021-04-13T06:47:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.