RelVAE: Generative Pretraining for few-shot Visual Relationship Detection

• URL: http://arxiv.org/abs/2311.16261v1
• Date: Mon, 27 Nov 2023 19:08:08 GMT
• Title: RelVAE: Generative Pretraining for few-shot Visual Relationship Detection
• Authors: Sotiris Karapiperis, Markos Diomataris, Vassilis Pitsikalis
• Abstract summary: We present the first pretraining method for few-shot predicate classification that does not require any annotated relations. We construct few-shot training splits and show quantitative experiments on VG200 and VRD datasets.
• Score: 2.2230760534775915
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual relations are complex, multimodal concepts that play an important role in the way humans perceive the world. As a result of their complexity, high-quality, diverse and large scale datasets for visual relations are still absent. In an attempt to overcome this data barrier, we choose to focus on the problem of few-shot Visual Relationship Detection (VRD), a setting that has been so far neglected by the community. In this work we present the first pretraining method for few-shot predicate classification that does not require any annotated relations. We achieve this by introducing a generative model that is able to capture the variation of semantic, visual and spatial information of relations inside a latent space and later exploiting its representations in order to achieve efficient few-shot classification. We construct few-shot training splits and show quantitative experiments on VG200 and VRD datasets where our model outperforms the baselines. Lastly we attempt to interpret the decisions of the model by conducting various qualitative experiments.

Related papers

• Generalized Visual Relation Detection with Diffusion Models [94.62313788626128]
  Visual relation detection (VRD) aims to identify relationships (or interactions) between object pairs in an image. We propose to model visual relations as continuous embeddings, and design diffusion models to achieve generalized VRD in a conditional generative manner. Our Diff-VRD is able to generate visual relations beyond the pre-defined category labels of datasets.
  arXiv  Detail & Related papers  (2025-04-16T14:03:24Z)
• Few-Shot, No Problem: Descriptive Continual Relation Extraction [27.296604792388646]
  Few-shot Continual Relation Extraction is a crucial challenge for enabling AI systems to identify and adapt to evolving relationships in real-world domains. Traditional memory-based approaches often overfit to limited samples, failing to reinforce old knowledge. We propose a novel retrieval-based solution, starting with a large language model to generate descriptions for each relation.
  arXiv  Detail & Related papers  (2025-02-27T23:44:30Z)
• Diversity Over Quantity: A Lesson From Few Shot Relation Classification [62.66895901654023]
  We show that training on a diverse set of relations significantly enhances a model's ability to generalize to unseen relations. We introduce REBEL-FS, a new FSRC benchmark that incorporates an order of magnitude more relation types than existing datasets.
  arXiv  Detail & Related papers  (2024-12-06T21:41:01Z)
• A Plug-and-Play Method for Rare Human-Object Interactions Detection by Bridging Domain Gap [50.079224604394]
  We present a novel model-agnostic framework called textbfContext-textbfEnhanced textbfFeature textbfAment (CEFA) CEFA consists of a feature alignment module and a context enhancement module. Our method can serve as a plug-and-play module to improve the detection performance of HOI models on rare categories.
  arXiv  Detail & Related papers  (2024-07-31T08:42:48Z)
• Corpus Considerations for Annotator Modeling and Scaling [9.263562546969695]
  We show that the commonly used user token model consistently outperforms more complex models. Our findings shed light on the relationship between corpus statistics and annotator modeling performance.
  arXiv  Detail & Related papers  (2024-04-02T22:27:24Z)
• Towards Human-Like Machine Comprehension: Few-Shot Relational Learning in Visually-Rich Documents [16.78371134590167]
  Key-value relations are prevalent in Visually-Rich Documents (VRDs) These non-textual cues serve as important indicators that greatly enhance human comprehension and acquisition of such relation triplets. Our research focuses on few-shot relational learning, specifically targeting the extraction of key-value relation triplets in VRDs.
  arXiv  Detail & Related papers  (2024-03-23T08:40:35Z)
• Unified Visual Relationship Detection with Vision and Language Models [89.77838890788638]
  This work focuses on training a single visual relationship detector predicting over the union of label spaces from multiple datasets. We propose UniVRD, a novel bottom-up method for Unified Visual Relationship Detection by leveraging vision and language models. Empirical results on both human-object interaction detection and scene-graph generation demonstrate the competitive performance of our model.
  arXiv  Detail & Related papers  (2023-03-16T00:06:28Z)
• Knowledge-augmented Few-shot Visual Relation Detection [25.457693302327637]
  Visual Relation Detection (VRD) aims to detect relationships between objects for image understanding. Most existing VRD methods rely on thousands of training samples of each relationship to achieve satisfactory performance. We devise a knowledge-augmented, few-shot VRD framework leveraging both textual knowledge and visual relation knowledge.
  arXiv  Detail & Related papers  (2023-03-09T15:38:40Z)
• Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations [78.12377360145078]
  Contrastive self-supervised learning has outperformed supervised pretraining on many downstream tasks like segmentation and object detection. In this paper, we first study how biases in the dataset affect existing methods. We show that current contrastive approaches work surprisingly well across: (i) object- versus scene-centric, (ii) uniform versus long-tailed and (iii) general versus domain-specific datasets.
  arXiv  Detail & Related papers  (2021-06-10T17:59:13Z)
• Few-shot Visual Reasoning with Meta-analogical Contrastive Learning [141.2562447971]
  We propose to solve a few-shot (or low-shot) visual reasoning problem, by resorting to analogical reasoning. We extract structural relationships between elements in both domains, and enforce them to be as similar as possible with analogical learning. We validate our method on RAVEN dataset, on which it outperforms state-of-the-art method, with larger gains when the training data is scarce.
  arXiv  Detail & Related papers  (2020-07-23T14:00:34Z)
• Relation-Guided Representation Learning [53.60351496449232]
  We propose a new representation learning method that explicitly models and leverages sample relations. Our framework well preserves the relations between samples. By seeking to embed samples into subspace, we show that our method can address the large-scale and out-of-sample problem.
  arXiv  Detail & Related papers  (2020-07-11T10:57:45Z)
• Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision [57.14468881854616]
  We propose an auxiliary training objective that improves the generalization capabilities of neural networks. We use pairs of minimally-different examples with different labels, a.k.a counterfactual or contrasting examples, which provide a signal indicative of the underlying causal structure of the task. Models trained with this technique demonstrate improved performance on out-of-distribution test sets.
  arXiv  Detail & Related papers  (2020-04-20T02:47:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.