Neural Slot Interpreters: Grounding Object Semantics in Emergent Slot Representations

• URL: http://arxiv.org/abs/2403.07887v1
• Date: Fri, 2 Feb 2024 12:37:23 GMT
• Title: Neural Slot Interpreters: Grounding Object Semantics in Emergent Slot Representations
• Authors: Bhishma Dedhia, Niraj K. Jha,
• Abstract summary: We present the Neural Slot Interpreter (NSI) that learns to ground and generate object semantics via slot representations. NSI is an XML-like programming language that uses simple syntax rules to organize the object semantics of a scene into object-centric program primitives.
• Score: 4.807052027638089
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Object-centric methods have seen significant progress in unsupervised decomposition of raw perception into rich object-like abstractions. However, limited ability to ground object semantics of the real world into the learned abstractions has hindered their adoption in downstream understanding applications. We present the Neural Slot Interpreter (NSI) that learns to ground and generate object semantics via slot representations. At the core of NSI is an XML-like programming language that uses simple syntax rules to organize the object semantics of a scene into object-centric program primitives. Then, an alignment model learns to ground program primitives into slots through a bi-level contrastive learning objective over a shared embedding space. Finally, we formulate the NSI program generator model to use the dense associations inferred from the alignment model to generate object-centric programs from slots. Experiments on bi-modal retrieval tasks demonstrate the efficacy of the learned alignments, surpassing set-matching-based predictors by a significant margin. Moreover, learning the program generator from grounded associations enhances the predictive power of slots. NSI generated programs demonstrate improved performance of object-centric learners on property prediction and object detection, and scale with real-world scene complexity.

Related papers

• What Makes a Maze Look Like a Maze? [92.80800000328277]
  We introduce Deep Grounding (DSG), a framework that leverages explicit structured representations of visual abstractions for grounding and reasoning. At the core of DSG are schemas--dependency graph descriptions of abstract concepts that decompose them into more primitive-level symbols. We show that DSG significantly improves the abstract visual reasoning performance of vision-language models.
  arXiv  Detail & Related papers  (2024-09-12T16:41:47Z)
• In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentation [50.79940712523551]
  We present lazy visual grounding, a two-stage approach of unsupervised object mask discovery followed by object grounding. Our model requires no additional training yet shows great performance on five public datasets.
  arXiv  Detail & Related papers  (2024-08-09T09:28:35Z)
• ZoomNeXt: A Unified Collaborative Pyramid Network for Camouflaged Object Detection [70.11264880907652]
  Recent object (COD) attempts to segment objects visually blended into their surroundings, which is extremely complex and difficult in real-world scenarios. We propose an effective unified collaborative pyramid network that mimics human behavior when observing vague images and camouflaged zooming in and out. Our framework consistently outperforms existing state-of-the-art methods in image and video COD benchmarks.
  arXiv  Detail & Related papers  (2023-10-31T06:11:23Z)
• Spotlight Attention: Robust Object-Centric Learning With a Spatial Locality Prior [88.9319150230121]
  Object-centric vision aims to construct an explicit representation of the objects in a scene. We incorporate a spatial-locality prior into state-of-the-art object-centric vision models. We obtain significant improvements in segmenting objects in both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-05-31T04:35:50Z)
• Hyperbolic Contrastive Learning for Visual Representations beyond Objects [30.618032825306187]
  We focus on learning representations for objects and scenes that preserve the structure among them. Motivated by the observation that visually similar objects are close in the representation space, we argue that the scenes and objects should instead follow a hierarchical structure.
  arXiv  Detail & Related papers  (2022-12-01T16:58:57Z)
• Visual Superordinate Abstraction for Robust Concept Learning [80.15940996821541]
  Concept learning constructs visual representations that are connected to linguistic semantics. We ascribe the bottleneck to a failure of exploring the intrinsic semantic hierarchy of visual concepts. We propose a visual superordinate abstraction framework for explicitly modeling semantic-aware visual subspaces.
  arXiv  Detail & Related papers  (2022-05-28T14:27:38Z)
• Sim-To-Real Transfer of Visual Grounding for Human-Aided Ambiguity Resolution [0.0]
  We consider the task of visual grounding, where the agent segments an object from a crowded scene given a natural language description. Modern holistic approaches to visual grounding usually ignore language structure and struggle to cover generic domains. We introduce a fully decoupled modular framework for compositional visual grounding of entities, attributes, and spatial relations.
  arXiv  Detail & Related papers  (2022-05-24T14:12:32Z)
• Constellation: Learning relational abstractions over objects for compositional imagination [64.99658940906917]
  We introduce Constellation, a network that learns relational abstractions of static visual scenes. This work is a first step in the explicit representation of visual relationships and using them for complex cognitive procedures.
  arXiv  Detail & Related papers  (2021-07-23T11:59:40Z)
• Object-Centric Learning with Slot Attention [43.684193749891506]
  We present the Slot Attention module, an architectural component that interfaces with perceptual representations. Slot Attention produces task-dependent abstract representations which we call slots. We empirically demonstrate that Slot Attention can extract object-centric representations that enable generalization to unseen compositions.
  arXiv  Detail & Related papers  (2020-06-26T15:31:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.