LASA: A Weak Supervision Method for Open-Vocabulary Scene Sketch Semantic Segmentation

• URL: http://arxiv.org/abs/2606.11837v1
• Date: Wed, 10 Jun 2026 09:17:36 GMT
• Title: LASA: A Weak Supervision Method for Open-Vocabulary Scene Sketch Semantic Segmentation
• Abstract summary: Open-vocabulary scene sketch semantic segmentation aims to assign dense semantic labels to sparse line drawings based on flexible category vocabularies specified at inference time.<n>Unlike natural images, sketches lack texture and color cues, making semantic understanding heavily dependent on stroke layout and spatial configuration.<n>We propose a structure-aware framework built upon textbfLayer-wise textbfAccumulated textbfStructural textbfAttention (textbfLASA), which aggregates multi-layer attention to guide hierarchical semantic alignment under weak supervision
• Score: 9.681252371247643
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-vocabulary scene sketch semantic segmentation aims to assign dense semantic labels to sparse line drawings based on flexible category vocabularies specified at inference time, without relying on pixel-level annotations during training. Unlike natural images, sketches lack texture and color cues, making semantic understanding heavily dependent on stroke layout and spatial configuration, a challenge that renders single-layer vision-language features inherently unstable. Our key observation is that attention maps from different Vision Transformer layers encode complementary spatial cues: shallow layers capture global structural layouts, while deeper layers focus on local stroke intersections and object parts. This suggests that cross-layer aggregation provides a more robust structural prior than any individual layer alone. Leveraging this insight, we propose a structure-aware framework built upon \textbf{L}ayer-wise \textbf{A}ccumulated \textbf{S}tructural \textbf{A}ttention (\textbf{LASA}), which aggregates multi-layer attention to guide hierarchical semantic alignment under weak supervision and refine predictions during inference. Experiments on FS-COCO, SFSD, and FrISS show that LASA improves mIoU by $+3.43$, $+8.01$, and $+15.74$ over the prior weakly supervised baselines, demonstrating consistent gains in both segmentation accuracy and spatial coherence. Our source code will be made publicly available.

Related papers

• Hierarchical Prompt Injector for Domain Generalization Segmentation [21.255611918751104]
  Domain Generalized Semantic (DGSS) is a challenging task, as vision models often rely on low-level appearance cues that change across domains.<n>Existing methods use prompt learning to transfer structural priors into DGSS models, but typically encode each class as a single holistic prompt.<n>We address this with textbfSpatial Hierarchical Prompts (SHP) that enrich each class with region-level geometric anchors capturing structural appearance from distinct viewing angles.
  arXiv  Detail & Related papers  (2026-09-05T03:56:00Z)
• GeoSeg-OV: Bridging Geospatial Gaps with Structural Guidance for Open-Vocabulary Remote Sensing Segmentation [7.3929592417688275]
  We propose GeoSeg-OV, which decouples auxiliary VFM features from visual-text matching and repurposes them as structural guidance for cost aggregation and decoding.<n>On the global High-Resolution Land Cover benchmark spanning seven datasets across six continents, GeoSeg-OV outperforms the state-of-the-art by +2.5 and +2.7 average mIoU under two training settings.
  arXiv  Detail & Related papers  (2026-08-11T03:18:22Z)
• TraceCLIP: Recovering Local Semantics from Patch-to-CLS Contributions [20.430455131220544]
  We introduce TraceCLIP, a training-free framework that recovers latent patch-level semantic evidence.<n>We show that TraceCLIP achieves gains of 1.3 to 4.5 points in average mIoU over the strongest prior training-free methods.
  arXiv  Detail & Related papers  (2026-07-28T11:25:27Z)
• DINOde: Continuous Vision-Text Alignment for Open-Vocabulary Semantic Segmentation [53.011370226020695]
  We propose DINOde, an ODE-based framework that aligns CLIP text embeddings with the DINO visual manifold.<n>Our approach employs two complementary components: (i) Semantic Text Flow (STF), which evolves text embeddings toward the DINO manifold through a continuous ODE trajectory, and (ii) Global Context Flow (GCF), which refines the holistic image representation carried by DINO's CLS token.<n>By modeling alignment as a continuous trajectory, DINOde avoids the entanglement inherent in discrete projections and yields more robust cross-modal alignment.
  arXiv  Detail & Related papers  (2026-07-23T14:37:27Z)
• OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following [59.53720386342017]
  We propose a unified vision-language framework that adapts foundational Vision-Language Models for scalable multi-person gaze reasoning.<n>By modeling all individuals, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning.
  arXiv  Detail & Related papers  (2026-05-26T00:08:06Z)
• dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3 [36.6036728217708]
  Open-Vocabulary Semantics (OVSS) assigns pixel-level labels from an open set of text-defined categories, demanding reliable generalization to unseen classes at inference.<n>We introduce dinov3.seg, extending dinov3.txt into a dedicated framework for OVSS.<n>Our contributions are four-fold. First, we design a task-specific architecture tailored to this backbone, systematically adapting established design principles from prior open-vocabulary segmentation work.<n>Second, we jointly leverage text embeddings aligned with both the global [] token and local patch-level visual features from ViT-based encoder
  arXiv  Detail & Related papers  (2026-03-19T23:57:28Z)
• PCA-Seg: Revisiting Cost Aggregation for Open-Vocabulary Semantic and Part Segmentation [58.1914505657064]
  This paper proposes a simple yet effective parallel cost aggregation (PCA-Seg) paradigm to alleviate the challenge of knowledge interference between class-level semantics and spatial context.<n>Experiments on eight benchmarks show that each parallel block in PCA-Seg adds merely 0.35M parameters while achieving state-of-the-art OSPS performance.
  arXiv  Detail & Related papers  (2026-03-18T09:26:43Z)
• Aligning Forest and Trees in Images and Long Captions for Visually Grounded Understanding [31.471221254091294]
  Large vision-language models such as CLIP struggle with long captions because they align images and texts as undifferentiated wholes.<n>We propose CAFT, a hierarchical image-text representation learning framework that aligns global and local semantics across images and long captions without pixel-level supervision.
  arXiv  Detail & Related papers  (2026-02-03T01:31:55Z)
• DiSa: Saliency-Aware Foreground-Background Disentangled Framework for Open-Vocabulary Semantic Segmentation [16.57245702815661]
  Open-vocabulary semantic segmentation aims to assign labels to every pixel in an image based on text labels.<n>Existing approaches typically utilize vision-language models (VLMs), such as CLIP, for dense prediction.<n>We introduce DiSa, a novel saliency-aware foreground-background disentangled framework.
  arXiv  Detail & Related papers  (2026-01-27T21:15:10Z)
• ESCA: Contextualizing Embodied Agents via Scene-Graph Generation [47.008144510161486]
  We propose ESCA, a framework that contextualizes embodied agents by grounding their perception in spatial-temporal scene graphs.<n>At its core is SGCLIP, a novel, open-domain, promptable foundation model for generating scene graphs.<n>SGCLIP excels in both prompt-based inference and task-specific fine-tuning, achieving state-of-the-art results on scene graph generation and action localization benchmarks.
  arXiv  Detail & Related papers  (2025-10-11T20:13:59Z)
• CoPatch: Zero-Shot Referring Image Segmentation by Leveraging Untapped Spatial Knowledge in CLIP [26.827036116024914]
  textscCoPatch is a zero-shot RIS framework that enhances spatial representations in both text and image modalities.<n>We show that textscCoPatch significantly improves spatial grounding in zero-shot RIS across RefCOCO, RefCOCO+, RefCOCOg, and PhraseCut (+ 2--7 mIoU) without requiring any additional training.
  arXiv  Detail & Related papers  (2025-09-27T04:12:10Z)
• Annotation-Free Open-Vocabulary Segmentation for Remote-Sensing Images [51.74614065919118]
  This paper introduces SegEarth-OV, the first framework for annotation-free open-vocabulary segmentation of RS images.<n>We propose SimFeatUp, a universal upsampler that robustly restores high-resolution spatial details from coarse features.<n>We also present a simple yet effective Global Bias Alleviation operation to subtract the inherent global context from patch features.
  arXiv  Detail & Related papers  (2025-08-25T14:22:57Z)
• Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception [71.26728044621458]
  DeCLIP is a novel framework that enhances CLIP by decoupling the self-attention module to obtain content'' and context'' features respectively.<n>It consistently achieves state-of-the-art performance across a broad spectrum of tasks, including 2D detection and segmentation, 3D instance segmentation, video instance segmentation, and 6D object pose estimation.
  arXiv  Detail & Related papers  (2025-08-15T06:43:51Z)
• N2F2: Hierarchical Scene Understanding with Nested Neural Feature Fields [112.02885337510716]
  Nested Neural Feature Fields (N2F2) is a novel approach that employs hierarchical supervision to learn a single feature field. We leverage a 2D class-agnostic segmentation model to provide semantically meaningful pixel groupings at arbitrary scales in the image space. Our approach outperforms the state-of-the-art feature field distillation methods on tasks such as open-vocabulary 3D segmentation and localization.
  arXiv  Detail & Related papers  (2024-03-16T18:50:44Z)
• Open-Vocabulary Camouflaged Object Segmentation [66.94945066779988]
  We introduce a new task, open-vocabulary camouflaged object segmentation (OVCOS) We construct a large-scale complex scene dataset (textbfOVCamo) containing 11,483 hand-selected images with fine annotations and corresponding object classes. By integrating the guidance of class semantic knowledge and the supplement of visual structure cues from the edge and depth information, the proposed method can efficiently capture camouflaged objects.
  arXiv  Detail & Related papers  (2023-11-19T06:00:39Z)
• Deep Hierarchical Semantic Segmentation [76.40565872257709]
  hierarchical semantic segmentation (HSS) aims at structured, pixel-wise description of visual observation in terms of a class hierarchy. HSSN casts HSS as a pixel-wise multi-label classification task, only bringing minimal architecture change to current segmentation models. With hierarchy-induced margin constraints, HSSN reshapes the pixel embedding space, so as to generate well-structured pixel representations.
  arXiv  Detail & Related papers  (2022-03-27T15:47:44Z)
• Unsupervised Semantic Segmentation by Distilling Feature Correspondences [94.73675308961944]
  Unsupervised semantic segmentation aims to discover and localize semantically meaningful categories within image corpora without any form of annotation. We present STEGO, a novel framework that distills unsupervised features into high-quality discrete semantic labels. STEGO yields a significant improvement over the prior state of the art, on both the CocoStuff and Cityscapes challenges.
  arXiv  Detail & Related papers  (2022-03-16T06:08:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.