Identifying concept libraries from language about object structure
- URL: http://arxiv.org/abs/2205.05666v1
- Date: Wed, 11 May 2022 17:49:25 GMT
- Title: Identifying concept libraries from language about object structure
- Authors: Catherine Wong, William P. McCarthy, Gabriel Grand, Yoni Friedman,
Joshua B. Tenenbaum, Jacob Andreas, Robert D. Hawkins, Judith E. Fan
- Abstract summary: We leverage natural language descriptions for a diverse set of 2K procedurally generated objects to identify the parts people use.
We formalize our problem as search over a space of program libraries that contain different part concepts.
By combining naturalistic language at scale with structured program representations, we discover a fundamental information-theoretic tradeoff governing the part concepts people name.
- Score: 56.83719358616503
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Our understanding of the visual world goes beyond naming objects,
encompassing our ability to parse objects into meaningful parts, attributes,
and relations. In this work, we leverage natural language descriptions for a
diverse set of 2K procedurally generated objects to identify the parts people
use and the principles leading these parts to be favored over others. We
formalize our problem as search over a space of program libraries that contain
different part concepts, using tools from machine translation to evaluate how
well programs expressed in each library align to human language. By combining
naturalistic language at scale with structured program representations, we
discover a fundamental information-theoretic tradeoff governing the part
concepts people name: people favor a lexicon that allows concise descriptions
of each object, while also minimizing the size of the lexicon itself.
Related papers
- SAGE: Bridging Semantic and Actionable Parts for GEneralizable Manipulation of Articulated Objects [9.500480417077272]
We propose a novel framework that bridges semantic and actionable parts of articulated objects to achieve generalizable manipulation under natural language instructions.
A part-grounding module maps the semantic parts into so-called Generalizable Actionable Parts (GAParts), which inherently carry information about part motion.
An interactive feedback module is incorporated to respond to failures, which closes the loop and increases the robustness of the overall framework.
arXiv Detail & Related papers (2023-12-03T07:22:42Z) - Comparative Analysis of Widely use Object-Oriented Languages [0.0]
Learning of object-oriented paradigm is compulsory in every computer science major.
It is difficult to choose which should be the first programming language in order to teach object-oriented principles.
arXiv Detail & Related papers (2023-06-02T12:28:13Z) - Embodied Concept Learner: Self-supervised Learning of Concepts and
Mapping through Instruction Following [101.55727845195969]
We propose Embodied Learner Concept (ECL) in an interactive 3D environment.
A robot agent can ground visual concepts, build semantic maps and plan actions to complete tasks.
ECL is fully transparent and step-by-step interpretable in long-term planning.
arXiv Detail & Related papers (2023-04-07T17:59:34Z) - Position-Aware Contrastive Alignment for Referring Image Segmentation [65.16214741785633]
We present a position-aware contrastive alignment network (PCAN) to enhance the alignment of multi-modal features.
Our PCAN consists of two modules: 1) Position Aware Module (PAM), which provides position information of all objects related to natural language descriptions, and 2) Contrastive Language Understanding Module (CLUM), which enhances multi-modal alignment.
arXiv Detail & Related papers (2022-12-27T09:13:19Z) - Differentiable Parsing and Visual Grounding of Verbal Instructions for
Object Placement [26.74189486483276]
We introduce ParaGon, a PARsing And visual GrOuNding framework for language-conditioned object placement.
It parses language instructions into relations between objects and grounds those objects in visual scenes.
ParaGon encodes all of those procedures into neural networks for end-to-end training.
arXiv Detail & Related papers (2022-10-01T07:36:51Z) - Leveraging Language to Learn Program Abstractions and Search Heuristics [66.28391181268645]
We introduce LAPS (Language for Abstraction and Program Search), a technique for using natural language annotations to guide joint learning of libraries and neurally-guided search models for synthesis.
When integrated into a state-of-the-art library learning system (DreamCoder), LAPS produces higher-quality libraries and improves search efficiency and generalization.
arXiv Detail & Related papers (2021-06-18T15:08:47Z) - Low-Dimensional Structure in the Space of Language Representations is
Reflected in Brain Responses [62.197912623223964]
We show a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings.
We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI.
This suggests that the embedding captures some part of the brain's natural language representation structure.
arXiv Detail & Related papers (2021-06-09T22:59:12Z) - Understanding Synonymous Referring Expressions via Contrastive Features [105.36814858748285]
We develop an end-to-end trainable framework to learn contrastive features on the image and object instance levels.
We conduct extensive experiments to evaluate the proposed algorithm on several benchmark datasets.
arXiv Detail & Related papers (2021-04-20T17:56:24Z) - Language-Mediated, Object-Centric Representation Learning [21.667413971464455]
We present Language-mediated, Object-centric Representation Learning (LORL)
LORL is a paradigm for learning disentangled, object-centric scene representations from vision and language.
It can be integrated with various unsupervised segmentation algorithms that are language-agnostic.
arXiv Detail & Related papers (2020-12-31T18:36:07Z) - Machine learning approach of Japanese composition scoring and writing
aided system's design [0.0]
A composition scoring system can greatly assist language learners.
It can make language leaner improve themselves in the process of output something.
Especially for foreign language learners, lexical and syntactic content are usually what they are more concerned about.
arXiv Detail & Related papers (2020-08-26T11:01:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.