Sample-Efficient Learning of Novel Visual Concepts
- URL: http://arxiv.org/abs/2306.09482v1
- Date: Thu, 15 Jun 2023 20:24:30 GMT
- Title: Sample-Efficient Learning of Novel Visual Concepts
- Authors: Sarthak Bhagat, Simon Stepputtis, Joseph Campbell, Katia Sycara
- Abstract summary: State-of-the-art deep learning models struggle to recognize novel objects in a few-shot setting.
We show that incorporating a symbolic knowledge graph into a state-of-the-art recognition model enables a new approach for effective few-shot classification.
- Score: 7.398195748292981
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Despite the advances made in visual object recognition, state-of-the-art deep
learning models struggle to effectively recognize novel objects in a few-shot
setting where only a limited number of examples are provided. Unlike humans who
excel at such tasks, these models often fail to leverage known relationships
between entities in order to draw conclusions about such objects. In this work,
we show that incorporating a symbolic knowledge graph into a state-of-the-art
recognition model enables a new approach for effective few-shot classification.
In our proposed neuro-symbolic architecture and training methodology, the
knowledge graph is augmented with additional relationships extracted from a
small set of examples, improving its ability to recognize novel objects by
considering the presence of interconnected entities. Unlike existing few-shot
classifiers, we show that this enables our model to incorporate not only
objects but also abstract concepts and affordances. The existence of the
knowledge graph also makes this approach amenable to interpretability through
analysis of the relationships contained within it. We empirically show that our
approach outperforms current state-of-the-art few-shot multi-label
classification methods on the COCO dataset and evaluate the addition of
abstract concepts and affordances on the Visual Genome dataset.
Related papers
- Zero-Shot Object-Centric Representation Learning [72.43369950684057]
We study current object-centric methods through the lens of zero-shot generalization.
We introduce a benchmark comprising eight different synthetic and real-world datasets.
We find that training on diverse real-world images improves transferability to unseen scenarios.
arXiv Detail & Related papers (2024-08-17T10:37:07Z) - Automatic Discovery of Visual Circuits [66.99553804855931]
We explore scalable methods for extracting the subgraph of a vision model's computational graph that underlies recognition of a specific visual concept.
We find that our approach extracts circuits that causally affect model output, and that editing these circuits can defend large pretrained models from adversarial attacks.
arXiv Detail & Related papers (2024-04-22T17:00:57Z) - One-Shot Open Affordance Learning with Foundation Models [54.15857111929812]
We introduce One-shot Open Affordance Learning (OOAL), where a model is trained with just one example per base object category.
We propose a vision-language framework with simple and effective designs that boost the alignment between visual features and affordance text embeddings.
Experiments on two affordance segmentation benchmarks show that the proposed method outperforms state-of-the-art models with less than 1% of the full training data.
arXiv Detail & Related papers (2023-11-29T16:23:06Z) - Semantic Information for Object Detection [0.0]
We introduce a novel method for extracting a knowledge graph from a dataset of images provided with instance-level annotations.
We investigate the effectiveness of knowledge-aware re-optimization on the Faster-RCNN and DETR object detection models.
arXiv Detail & Related papers (2023-08-17T13:53:29Z) - Recognizing Unseen Objects via Multimodal Intensive Knowledge Graph
Propagation [68.13453771001522]
We propose a multimodal intensive ZSL framework that matches regions of images with corresponding semantic embeddings.
We conduct extensive experiments and evaluate our model on large-scale real-world data.
arXiv Detail & Related papers (2023-06-14T13:07:48Z) - Spatial Reasoning for Few-Shot Object Detection [21.3564383157159]
We propose a spatial reasoning framework that detects novel objects with only a few training examples in a context.
We employ a graph convolutional network as the RoIs and their relatedness are defined as nodes and edges, respectively.
We demonstrate that the proposed method significantly outperforms the state-of-the-art methods and verify its efficacy through extensive ablation studies.
arXiv Detail & Related papers (2022-11-02T12:38:08Z) - Unified Graph Structured Models for Video Understanding [93.72081456202672]
We propose a message passing graph neural network that explicitly models relational-temporal relations.
We show how our method is able to more effectively model relationships between relevant entities in the scene.
arXiv Detail & Related papers (2021-03-29T14:37:35Z) - Zero-Shot Human-Object Interaction Recognition via Affordance Graphs [3.867143522757309]
We propose a new approach for Zero-Shot Human-Object Interaction Recognition.
Our approach makes use of knowledge external to the image content in the form of a graph.
We evaluate our approach on several datasets and show that it outperforms the current state of the art.
arXiv Detail & Related papers (2020-09-02T13:14:44Z) - All About Knowledge Graphs for Actions [82.39684757372075]
We propose a better understanding of knowledge graphs (KGs) that can be utilized for zero-shot and few-shot action recognition.
We study three different construction mechanisms for KGs: action embeddings, action-object embeddings, visual embeddings.
We present extensive analysis of the impact of different KGs on different experimental setups.
arXiv Detail & Related papers (2020-08-28T01:44:01Z) - Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and
Novel-View Synthesis [39.53519330457627]
We propose a novel task of joint few-shot recognition and novel-view synthesis.
We aim to simultaneously learn an object classifier and generate images of that type of object from new viewpoints.
We focus on the interaction and cooperation between a generative model and a discriminative model.
arXiv Detail & Related papers (2020-08-16T19:40:56Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.