One-Shot Open Affordance Learning with Foundation Models

• URL: http://arxiv.org/abs/2311.17776v1
• Date: Wed, 29 Nov 2023 16:23:06 GMT
• Title: One-Shot Open Affordance Learning with Foundation Models
• Authors: Gen Li, Deqing Sun, Laura Sevilla-Lara, Varun Jampani
• Abstract summary: We introduce One-shot Open Affordance Learning (OOAL), where a model is trained with just one example per base object category. We propose a vision-language framework with simple and effective designs that boost the alignment between visual features and affordance text embeddings. Experiments on two affordance segmentation benchmarks show that the proposed method outperforms state-of-the-art models with less than 1% of the full training data.
• Score: 54.15857111929812
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce One-shot Open Affordance Learning (OOAL), where a model is trained with just one example per base object category, but is expected to identify novel objects and affordances. While vision-language models excel at recognizing novel objects and scenes, they often struggle to understand finer levels of granularity such as affordances. To handle this issue, we conduct a comprehensive analysis of existing foundation models, to explore their inherent understanding of affordances and assess the potential for data-limited affordance learning. We then propose a vision-language framework with simple and effective designs that boost the alignment between visual features and affordance text embeddings. Experiments on two affordance segmentation benchmarks show that the proposed method outperforms state-of-the-art models with less than 1% of the full training data, and exhibits reasonable generalization capability on unseen objects and affordances.

Related papers

• Few-shot target-driven instance detection based on open-vocabulary object detection models [1.0749601922718608]
  Open-vocabulary object detection models bring closer visual and textual concepts in the same latent space. We propose a lightweight method to turn the latter into a one-shot or few-shot object recognition models without requiring textual descriptions.
  arXiv  Detail & Related papers  (2024-10-21T14:03:15Z)
• High-Performance Few-Shot Segmentation with Foundation Models: An Empirical Study [64.06777376676513]
  We develop a few-shot segmentation (FSS) framework based on foundation models. To be specific, we propose a simple approach to extract implicit knowledge from foundation models to construct coarse correspondence. Experiments on two widely used datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-09-10T08:04:11Z)
• Envisioning Class Entity Reasoning by Large Language Models for Few-shot Learning [13.68867780184022]
  Few-shot learning aims to recognize new concepts using a limited number of visual samples. Our framework incorporates both the abstract class semantics and the concrete class entities extracted from Large Language Models (LLMs) For the challenging one-shot setting, our approach, utilizing the ResNet-12 backbone, achieves an average improvement of 1.95% over the second-best competitor.
  arXiv  Detail & Related papers  (2024-08-22T15:10:20Z)
• Zero-Shot Object-Centric Representation Learning [72.43369950684057]
  We study current object-centric methods through the lens of zero-shot generalization. We introduce a benchmark comprising eight different synthetic and real-world datasets. We find that training on diverse real-world images improves transferability to unseen scenarios.
  arXiv  Detail & Related papers  (2024-08-17T10:37:07Z)
• Distilling Vision-Language Foundation Models: A Data-Free Approach via Prompt Diversification [49.41632476658246]
  We discuss the extension of DFKD to Vision-Language Foundation Models without access to the billion-level image-text datasets. The objective is to customize a student model for distribution-agnostic downstream tasks with given category concepts. We propose three novel Prompt Diversification methods to encourage image synthesis with diverse styles.
  arXiv  Detail & Related papers  (2024-07-21T13:26:30Z)
• Few Shot Class Incremental Learning using Vision-Language models [24.930246674021525]
  In this study, we introduce an innovative few-shot class incremental learning (FSCIL) framework that utilizes language regularizer and subspace regularizer. Our proposed framework not only empowers the model to embrace novel classes with limited data, but also ensures the preservation of performance on base classes.
  arXiv  Detail & Related papers  (2024-05-02T06:52:49Z)
• Low-shot Object Learning with Mutual Exclusivity Bias [27.67152913041082]
  This paper introduces Low-shot Object Learning with Mutual Exclusivity Bias (LSME), the first computational framing of mutual exclusivity bias. We provide a novel dataset, comprehensive baselines, and a state-of-the-art method to enable the ML community to tackle this challenging learning task.
  arXiv  Detail & Related papers  (2023-12-06T14:54:10Z)
• Foundational Models Defining a New Era in Vision: A Survey and Outlook [151.49434496615427]
  Vision systems to see and reason about the compositional nature of visual scenes are fundamental to understanding our world. The models learned to bridge the gap between such modalities coupled with large-scale training data facilitate contextual reasoning, generalization, and prompt capabilities at test time. The output of such models can be modified through human-provided prompts without retraining, e.g., segmenting a particular object by providing a bounding box, having interactive dialogues by asking questions about an image or video scene or manipulating the robot's behavior through language instructions.
  arXiv  Detail & Related papers  (2023-07-25T17:59:18Z)
• An Empirical Investigation of Representation Learning for Imitation [76.48784376425911]
  Recent work in vision, reinforcement learning, and NLP has shown that auxiliary representation learning objectives can reduce the need for large amounts of expensive, task-specific data. We propose a modular framework for constructing representation learning algorithms, then use our framework to evaluate the utility of representation learning for imitation.
  arXiv  Detail & Related papers  (2022-05-16T11:23:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.