Revisiting Weakly-Supervised Video Scene Graph Generation via Pair Affinity Learning

• URL: http://arxiv.org/abs/2603.21559v1
• Date: Mon, 23 Mar 2026 04:28:51 GMT
• Title: Revisiting Weakly-Supervised Video Scene Graph Generation via Pair Affinity Learning
• Abstract summary: Weakly-supervised video scene graph generation (WS-VSGG) aims to parse video content into structured relational triplets without bounding box annotations and with only sparse temporal labeling.<n>We introduce a learnable pair affinity that estimates the likelihood of interaction between subject-object pairs.<n>Our approach consistently yields substantial improvements across different baselines and backbones, achieving state-of-the-art WS-VSGG performance.
• Score: 33.42009265471664
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Weakly-supervised video scene graph generation (WS-VSGG) aims to parse video content into structured relational triplets without bounding box annotations and with only sparse temporal labeling, significantly reducing annotation costs. Without ground-truth bounding boxes, these methods rely on off-the-shelf detectors to generate object proposals, yet largely overlook a fundamental discrepancy from fullysupervised pipelines. Fully-supervised detectors implicitly filter out noninteractive objects, while off-the-shelf detectors indiscriminately detect all visible objects, overwhelming relation models with noisy pairs.We address this by introducing a learnable pair affinity that estimates the likelihood of interaction between subject-object pairs. Through Pair Affinity Learning and Scoring (PALS), pair affinity is incorporated into inferencetime ranking and further integrated into contextual reasoning through Pair Affinity Modulation (PAM), enabling the model to suppress noninteractive pairs and focus on relationally meaningful ones. To provide cleaner supervision for pair affinity learning, we further propose Relation- Aware Matching (RAM), which leverages vision-language grounding to resolve class-level ambiguity in pseudo-label generation. Extensive experiments on Action Genome demonstrate that our approach consistently yields substantial improvements across different baselines and backbones, achieving state-of-the-art WS-VSGG performance.

Related papers

• Unveiling the Unknown: Open Vocabulary Object Detection with Scene Graphs [16.874298364235326]
  Open-vocabulary object detection seeks to identify novel object categories that were not part of the training data.<n>These methods often overlook structured, image-specific relationships between objects, such as interactions and spatial arrangements.<n>We propose a Scene-guided Modeling detection framework to address this issue.
  arXiv  Detail & Related papers  (2026-06-04T09:23:08Z)
• COVTrack++: Learning Open-Vocabulary Multi-Object Tracking from Continuous Videos via a Synergistic Paradigm [59.26203051651017]
  C-TAO is the first continuously annotated training set for Open-Vocabulary Multi-Object Tracking (OVMOT)<n>For the framework bottleneck, we propose COVTrack++, a synergistic framework that achieves a bidirectional reciprocal mechanism between detection and association through three modules.<n>Experiments on TAO demonstrate state-of-the-art performance, with novel TETA reaching 35.4% and 30.5% on validation and test sets, improving novel AssocA by 4.8% and novel LocA by 5.8% over previous methods.
  arXiv  Detail & Related papers  (2026-03-25T07:20:27Z)
• Interaction-Centric Knowledge Infusion and Transfer for Open-Vocabulary Scene Graph Generation [20.867572814544836]
  Open-vocabulary scene graph generation (OVSGG) extends traditional SGG by recognizing novel objects and beyond predefined relationships categories.<n>We propose an intertextbfACtion-textbfCentric end-to-end OVSGG framework (textbfACC) in an interaction-driven paradigm to minimize these mismatches.
  arXiv  Detail & Related papers  (2025-11-08T08:59:09Z)
• Imputation-free and Alignment-free: Incomplete Multi-view Clustering Driven by Consensus Semantic Learning [65.75756724642932]
  In incomplete multi-view clustering, missing data induce prototype shifts within views and semantic inconsistencies across views.<n>We propose an IMVC framework, imputation- and alignment-free for consensus semantics learning (FreeCSL)<n>FreeCSL achieves more confident and robust assignments on IMVC task, compared to state-of-the-art competitors.
  arXiv  Detail & Related papers  (2025-05-16T12:37:10Z)
• CoSD: Collaborative Stance Detection with Contrastive Heterogeneous Topic Graph Learning [18.75039816544345]
  We present a novel collaborative stance detection framework called (CoSD) CoSD learns topic-aware semantics and collaborative signals among texts, topics, and stance labels. Experiments on two benchmark datasets demonstrate the state-of-the-art detection performance of CoSD.
  arXiv  Detail & Related papers  (2024-04-26T02:04:05Z)
• Unified Visual Relationship Detection with Vision and Language Models [89.77838890788638]
  This work focuses on training a single visual relationship detector predicting over the union of label spaces from multiple datasets. We propose UniVRD, a novel bottom-up method for Unified Visual Relationship Detection by leveraging vision and language models. Empirical results on both human-object interaction detection and scene-graph generation demonstrate the competitive performance of our model.
  arXiv  Detail & Related papers  (2023-03-16T00:06:28Z)
• Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation [62.96628432641806]
  Scene Graph Generation aims to first encode the visual contents within the given image and then parse them into a compact summary graph. We first present a novel Stacked Hybrid-Attention network, which facilitates the intra-modal refinement as well as the inter-modal interaction. We then devise an innovative Group Collaborative Learning strategy to optimize the decoder.
  arXiv  Detail & Related papers  (2022-03-18T09:14:13Z)
• Modelling Neighbor Relation in Joint Space-Time Graph for Video Correspondence Learning [53.74240452117145]
  This paper presents a self-supervised method for learning reliable visual correspondence from unlabeled videos. We formulate the correspondence as finding paths in a joint space-time graph, where nodes are grid patches sampled from frames, and are linked by two types of edges. Our learned representation outperforms the state-of-the-art self-supervised methods on a variety of visual tasks.
  arXiv  Detail & Related papers  (2021-09-28T05:40:01Z)
• A Graph-based Interactive Reasoning for Human-Object Interaction Detection [71.50535113279551]
  We present a novel graph-based interactive reasoning model called Interactive Graph (abbr. in-Graph) to infer HOIs. We construct a new framework to assemble in-Graph models for detecting HOIs, namely in-GraphNet. Our framework is end-to-end trainable and free from costly annotations like human pose.
  arXiv  Detail & Related papers  (2020-07-14T09:29:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.