Constrained Structure Learning for Scene Graph Generation

• URL: http://arxiv.org/abs/2201.11697v1
• Date: Thu, 27 Jan 2022 17:47:37 GMT
• Title: Constrained Structure Learning for Scene Graph Generation
• Authors: Daqi Liu, Miroslaw Bober, Josef Kittler
• Abstract summary: We present a constrained structure learning method, for which an explicit constrained variational inference objective is proposed. We validate the proposed generic model on various popular scene graph generation benchmarks and show that it outperforms the state-of-the-art methods.
• Score: 40.46394569128303
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As a structured prediction task, scene graph generation aims to build a visually-grounded scene graph to explicitly model objects and their relationships in an input image. Currently, the mean field variational Bayesian framework is the de facto methodology used by the existing methods, in which the unconstrained inference step is often implemented by a message passing neural network. However, such formulation fails to explore other inference strategies, and largely ignores the more general constrained optimization models. In this paper, we present a constrained structure learning method, for which an explicit constrained variational inference objective is proposed. Instead of applying the ubiquitous message-passing strategy, a generic constrained optimization method - entropic mirror descent - is utilized to solve the constrained variational inference step. We validate the proposed generic model on various popular scene graph generation benchmarks and show that it outperforms the state-of-the-art methods.

Related papers

• Generative Modelling of Structurally Constrained Graphs [44.28728853270471]
  We present ConStruct, a novel framework that allows for hard-constraining graph diffusion models to incorporate specific properties, such as planarity or acyclicity. Our approach ensures that the sampled graphs remain within the domain of graphs that verify the specified property throughout the entire trajectory in both the forward and reverse processes.
  arXiv  Detail & Related papers  (2024-06-25T07:54:32Z)
• Robust Data-driven Prescriptiveness Optimization [4.792851066169871]
  This paper introduces a distributionally robust contextual optimization model where the coefficient of prescriptiveness substitutes for the classical empirical risk objective minimization. We evaluate the robustness of the resulting policies against alternative methods when the out-of-sample dataset is subject to varying amounts of distribution shift.
  arXiv  Detail & Related papers  (2023-06-09T14:56:06Z)
• Diffusion Action Segmentation [63.061058214427085]
  We propose a novel framework via denoising diffusion models, which shares the same inherent spirit of such iterative refinement. In this framework, action predictions are iteratively generated from random noise with input video features as conditions.
  arXiv  Detail & Related papers  (2023-03-31T10:53:24Z)
• Iterative Scene Graph Generation [55.893695946885174]
  Scene graph generation involves identifying object entities and their corresponding interaction predicates in a given image (or video) Existing approaches to scene graph generation assume certain factorization of the joint distribution to make the estimation iteration feasible. We propose a novel framework that addresses this limitation, as well as introduces dynamic conditioning on the image.
  arXiv  Detail & Related papers  (2022-07-27T10:37:29Z)
• Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation [88.14365009076907]
  Iterative refinement is a useful paradigm for representation learning. We develop an implicit differentiation approach that improves the stability and tractability of training.
  arXiv  Detail & Related papers  (2022-07-02T10:00:35Z)
• Doubly Reparameterized Importance Weighted Structure Learning for Scene Graph Generation [40.46394569128303]
  Scene graph generation, given an input image, aims to explicitly model objects and their relationships by constructing a visually-grounded scene graph. We propose a novel doubly re parameterized importance weighted structure learning method, which employs a tighter importance weighted lower bound as the variational inference objective. The proposed method achieves the state-of-the-art performance on various popular scene graph generation benchmarks.
  arXiv  Detail & Related papers  (2022-06-22T20:00:25Z)
• Importance Weighted Structure Learning for Scene Graph Generation [40.46394569128303]
  We propose a novel importance weighted structure learning method for scene graph generation. A generic entropic mirror descent algorithm is applied to solve the resulting constrained variational inference task. The proposed method achieves the state-of-the-art performance on various popular scene graph generation benchmarks.
  arXiv  Detail & Related papers  (2022-05-14T09:25:14Z)
• Bayesian Graph Contrastive Learning [55.36652660268726]
  We propose a novel perspective of graph contrastive learning methods showing random augmentations leads to encoders. Our proposed method represents each node by a distribution in the latent space in contrast to existing techniques which embed each node to a deterministic vector. We show a considerable improvement in performance compared to existing state-of-the-art methods on several benchmark datasets.
  arXiv  Detail & Related papers  (2021-12-15T01:45:32Z)
• Attentional Prototype Inference for Few-Shot Segmentation [128.45753577331422]
  We propose attentional prototype inference (API), a probabilistic latent variable framework for few-shot segmentation. We define a global latent variable to represent the prototype of each object category, which we model as a probabilistic distribution. We conduct extensive experiments on four benchmarks, where our proposal obtains at least competitive and often better performance than state-of-the-art prototype-based methods.
  arXiv  Detail & Related papers  (2021-05-14T06:58:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.