Learning to Assemble Novel Structures with Unfamiliar Parts under Semantic Constraints

• URL: http://arxiv.org/abs/2608.13684v1
• Date: Thu, 13 Aug 2026 18:27:29 GMT
• Title: Learning to Assemble Novel Structures with Unfamiliar Parts under Semantic Constraints
• Abstract summary: We focus on scenarios where an agent encounters, after deployment, semantic constraints on structures--in other words, constraints as to which part types and features make valid structures.<n>We study this setting in a simulated toy truck assembly domain, learning from symbolic evidence encoded in natural language and from dense visual observations.
• Score: 14.822812310776948
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper describes a neurosymbolic architecture for learning to assemble novel structures using evidence from embodied conversations and task demonstrations. We focus on scenarios where an agent encounters, after deployment, semantic constraints on structures--in other words, constraints as to which part types and features make valid structures--that were not available during training, and where it is initially unaware of the relevant structure and component part concepts. The agent must acquire and exploit such knowledge through user interactions while attempting assembly. We study this setting in a simulated toy truck assembly domain, learning from symbolic evidence encoded in natural language and from dense visual observations. Our experiments show that communicating semantic constraints through natural language (e.g., "dump trucks have a dumper") yields more data-efficient online adaptation than relying only on task demonstrations and/or only naming the parts through natural language.

Related papers

• SkeletonContext: Skeleton-side Context Prompt Learning for Zero-Shot Skeleton-based Action Recognition [22.87993245982946]
  Zero-shot skeleton-based action recognition aims to recognize unseen actions by transferring knowledge from seen categories through semantic descriptions.<n>Most existing methods typically align skeleton features with textual embeddings within a shared latent space.<n>We propose SkeletonContext, a prompt-based framework that enriches skeletal motion representations with language-driven contextual semantics.
  arXiv  Detail & Related papers  (2026-03-31T12:46:21Z)
• On the Emergence and Test-Time Use of Structural Information in Large Language Models [52.28603345019514]
  We study how language models learn abstract structures and utilize the learnt structural information at test-time.<n>We empirically show that the emergence of learning structural information correlates with complex reasoning tasks.
  arXiv  Detail & Related papers  (2026-01-25T15:02:25Z)
• Learning Compositional Behaviors from Demonstration and Language [28.352574199884852]
  BLADE is a framework for long-horizon robotic manipulation by integrating imitation learning and model-based planning.<n>We show significant capabilities in generalizing to novel situations, including novel initial states, external state perturbations, and novel goals.
  arXiv  Detail & Related papers  (2025-05-28T05:19:59Z)
• Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognition [50.86415025650168]
  Masked image modeling (MIM) tends to exploit local structures to reconstruct visual patterns, resulting in limited linguistic knowledge.<n>We propose a Linguistics-aware Masked Image Modeling (LMIM) approach, which channels the linguistic information into the decoding process of MIM through a separate branch.
  arXiv  Detail & Related papers  (2025-03-24T14:53:35Z)
• Punctuation Restoration Improves Structure Understanding Without Supervision [5.925894224649895]
  We show that punctuation restoration as a learning objective improves performance on structure-related tasks.<n>Our results show that punctuation restoration is an effective learning objective that can improve structure understanding.
  arXiv  Detail & Related papers  (2024-02-13T11:22:52Z)
• DeepStruct: Pretraining of Language Models for Structure Prediction [64.84144849119554]
  We pretrain language models on a collection of task-agnostic corpora to generate structures from text. Our structure pretraining enables zero-shot transfer of the learned knowledge that models have about the structure tasks. We show that a 10B parameter language model transfers non-trivially to most tasks and obtains state-of-the-art performance on 21 of 28 datasets.
  arXiv  Detail & Related papers  (2022-05-21T00:58:22Z)
• Dependency Induction Through the Lens of Visual Perception [81.91502968815746]
  We propose an unsupervised grammar induction model that leverages word concreteness and a structural vision-based to jointly learn constituency-structure and dependency-structure grammars. Our experiments show that the proposed extension outperforms the current state-of-the-art visually grounded models in constituency parsing even with a smaller grammar size.
  arXiv  Detail & Related papers  (2021-09-20T18:40:37Z)
• Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses [62.197912623223964]
  We show a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings. We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI. This suggests that the embedding captures some part of the brain's natural language representation structure.
  arXiv  Detail & Related papers  (2021-06-09T22:59:12Z)
• Modelling Compositionality and Structure Dependence in Natural Language [0.12183405753834563]
  Drawing on linguistics and set theory, a formalisation of these ideas is presented in the first half of this thesis. We see how cognitive systems that process language need to have certain functional constraints. Using the advances of word embedding techniques, a model of relational learning is simulated.
  arXiv  Detail & Related papers  (2020-11-22T17:28:50Z)
• Unsupervised Distillation of Syntactic Information from Contextualized Word Representations [62.230491683411536]
  We tackle the task of unsupervised disentanglement between semantics and structure in neural language representations. To this end, we automatically generate groups of sentences which are structurally similar but semantically different. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics.
  arXiv  Detail & Related papers  (2020-10-11T15:13:18Z)
• Visually Analyzing Contextualized Embeddings [2.802183323381949]
  We introduce a method for visually analyzing contextualized embeddings produced by deep neural network-based language models. Our approach is inspired by linguistic probes for natural language processing, where tasks are designed to probe language models for linguistic structure.
  arXiv  Detail & Related papers  (2020-09-05T15:40:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.