From Bounding Boxes to Visual Reasoning: An On-Policy Data Annotation Tool for Vision-Language Models

• URL: http://arxiv.org/abs/2606.18846v1
• Date: Wed, 17 Jun 2026 09:25:59 GMT
• Title: From Bounding Boxes to Visual Reasoning: An On-Policy Data Annotation Tool for Vision-Language Models
• Abstract summary: Vision-language models (VLMs) are rapidly advancing toward sophisticated grounded structured visual reasoning.<n>Existing data annotation tools fail to meet these intricate demands.<n>We introduce an open-source annotation tool, ScreenAnnotator.
• Score: 12.46500416733569
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) are rapidly advancing toward sophisticated grounded structured visual reasoning. Training models for such advanced capabilities demands a new genre of data that seamlessly unifies spatial coordinates, open-vocabulary descriptions, structured attributes, and topological relationships into a singular representation. However, existing data annotation tools fundamentally fail to meet these intricate demands, suffering from three systematic bottlenecks: limited expressiveness, severe annotation-training decoupling, and poor data reusability. To bridge this infrastructure gap, we introduce an open-source annotation tool, ScreenAnnotator. First, we define a unified annotation atom schema that binds spatial, semantic, and structural primitives into a single unit. Second, we implement an on-policy annotation loop embedded with a Bayesian Annotation Verifier (BAV). Finally, we design a template-driven multi-task data synthesis process dynamically transforms static atoms into diverse multi-dimensional reasoning tasks, eliminating redundant re-annotation. The on-policy loop drives the annotation accept rate to nearly 100% on flowcharts and 77% on GUI screenshots, while steadily reducing per-image annotation time as labeled data accumulate. In the flowchart scenario, fine-tuning a VLM yields 76.1% average accuracy, which is a 35.1% point absolute gain. Our code is available at: https://github.com/WnQinm/Annotator.

Related papers

• Unified Video Dense Prediction from Disjoint Data [71.9957801898161]
  We introduce UniD, a unified video model that jointly predicts eight dense scene properties-depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials.<n>We propose a simple yet effective distillation step in which per-task experts supervise a unified backbone through lightweight task projectors.<n>UniD achieves competitive performance against per-task specialists and multi-task baselines.
  arXiv  Detail & Related papers  (2026-07-23T17:59:50Z)
• Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning [27.48318501076437]
  Vision-Language Models (VLMs) often yield inconsistent descriptions of the same object across viewpoints.<n>We introduce a unified, memory-augmented Vision-Language agent that simultaneously handles data association, object captioning, and exploration policy.<n>We demonstrate improvements of up to +11.86% in standard captioning scores and +7.39% in caption self-similarity over baseline models.
  arXiv  Detail & Related papers  (2026-03-25T12:52:32Z)
• RefAtomNet++: Advancing Referring Atomic Video Action Recognition using Semantic Retrieval based Multi-Trajectory Mamba [86.47790050206306]
  RefAVA++ comprises >2.9 million frames and >75.1k annotated persons in total.<n> RefAtomNet++ advances cross-modal token aggregation through a multi-hierarchical semantic-aligned cross-attention mechanism.<n>Experiments show that RefAtomNet++ establishes new state-of-the-art results.
  arXiv  Detail & Related papers  (2025-10-18T10:41:19Z)
• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation [120.23172120151821]
  We propose a novel approach for disentangling visual and semantic features from the backbones of pre-trained diffusion models.<n>We introduce an automated pipeline that constructs image pairs with annotated semantic and visual correspondences.<n>We propose a new metric, Visual Semantic Matching, that quantifies visual inconsistencies in subject-driven image generation.
  arXiv  Detail & Related papers  (2025-09-26T07:11:55Z)
• Open-Vocabulary Octree-Graph for 3D Scene Understanding [54.11828083068082]
  Octree-Graph is a novel scene representation for open-vocabulary 3D scene understanding. An adaptive-octree structure is developed that stores semantics and depicts the occupancy of an object adjustably according to its shape.
  arXiv  Detail & Related papers  (2024-11-25T10:14:10Z)
• Harlequin: Color-driven Generation of Synthetic Data for Referring Expression Comprehension [4.164728134421114]
  Referring Expression (REC) aims to identify a particular object in a scene by a natural language expression, and is an important topic in visual language understanding. State-of-the-art methods for this task are based on deep learning, which generally requires expensive and manually labeled annotations. We propose a novel framework that generates artificial data for the REC task, taking into account both textual and visual modalities.
  arXiv  Detail & Related papers  (2024-11-22T09:08:36Z)
• ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding [42.10086029931937]
  Visual grounding aims to localize the object referred to in an image based on a natural language query. Existing methods demonstrate a significant performance drop when there are multiple distractions in an image. We propose a novel approach, the Relation and Semantic-sensitive Visual Grounding (ResVG) model, to address this issue.
  arXiv  Detail & Related papers  (2024-08-29T07:32:01Z)
• Open-Vocabulary Camouflaged Object Segmentation [66.94945066779988]
  We introduce a new task, open-vocabulary camouflaged object segmentation (OVCOS) We construct a large-scale complex scene dataset (textbfOVCamo) containing 11,483 hand-selected images with fine annotations and corresponding object classes. By integrating the guidance of class semantic knowledge and the supplement of visual structure cues from the edge and depth information, the proposed method can efficiently capture camouflaged objects.
  arXiv  Detail & Related papers  (2023-11-19T06:00:39Z)
• Dynamic Graph Message Passing Networks for Visual Recognition [112.49513303433606]
  Modelling long-range dependencies is critical for scene understanding tasks in computer vision. A fully-connected graph is beneficial for such modelling, but its computational overhead is prohibitive. We propose a dynamic graph message passing network, that significantly reduces the computational complexity.
  arXiv  Detail & Related papers  (2022-09-20T14:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.