Reusable Slotwise Mechanisms

• URL: http://arxiv.org/abs/2302.10503v2
• Date: Fri, 27 Oct 2023 07:33:22 GMT
• Title: Reusable Slotwise Mechanisms
• Abstract summary: We introduce Reusable Slotwise Mechanisms, or RSM, a framework that models object dynamics by leveraging communication among slots. RSM leverages the Central Contextual Information (CCI), enabling selected mechanisms to access the remaining slots. Experimental results demonstrate the superior performance of RSM compared to state-of-the-art methods.
• Score: 45.48002329758044
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agents with the ability to comprehend and reason about the dynamics of objects would be expected to exhibit improved robustness and generalization in novel scenarios. However, achieving this capability necessitates not only an effective scene representation but also an understanding of the mechanisms governing interactions among object subsets. Recent studies have made significant progress in representing scenes using object slots. In this work, we introduce Reusable Slotwise Mechanisms, or RSM, a framework that models object dynamics by leveraging communication among slots along with a modular architecture capable of dynamically selecting reusable mechanisms for predicting the future states of each object slot. Crucially, RSM leverages the Central Contextual Information (CCI), enabling selected mechanisms to access the remaining slots through a bottleneck, effectively allowing for modeling of higher order and complex interactions that might require a sparse subset of objects. Experimental results demonstrate the superior performance of RSM compared to state-of-the-art methods across various future prediction and related downstream tasks, including Visual Question Answering and action planning. Furthermore, we showcase RSM's Out-of-Distribution generalization ability to handle scenes in intricate scenarios.

Related papers

• Beyond Instance Slots: Semantically Rich World Models for Physical Interaction Planning [3.284163275977256]
  We present the Semantically Rich World Model (SR-WM), a task-conditioned world model structured around five functional roles.<n>Within SR-WM, a visual entity encoder extracts soft entity hypotheses from pretrained patch features, allowing segmentation masks to serve as optional proposal priors.<n>A role binder maps these hypotheses to task-specific roles, while an action conditioned dynamics model predicts role transitions alongside fine-grained semantics.
  arXiv  Detail & Related papers  (2026-08-23T08:37:59Z)
• From World Action Models to Embodied Brains: A Roadmap for Open-World Physical Intelligence [54.62791372989525]
  Action models, vision-language--action policies, and world models have advanced this goal.<n>World Action Models (WAMs) are particularly promising because they connect candidate interventions with predicted consequences.<n>We review the evolution toward WAMs and organize these limitations into three coupled gaps: model roles and representations, objectives and standardization, and system composition.
  arXiv  Detail & Related papers  (2026-07-13T15:22:56Z)
• Decoupling the Declarative from the Procedural in Vision-Language-Action Models [65.22639791239932]
  Policy trained to clone a behavior from object-specific demonstrations must generalize beyond that object.<n>We propose w$2$VLA, a new Vision-Language-Action model with restructured information flow.<n>Unlike popular, state-of-the-art VLAs, we show that our modular approach successfully decouples knowledge representations.
  arXiv  Detail & Related papers  (2026-06-19T14:43:58Z)
• DIML: Differentiable Inverse Mechanism Learning from Behaviors of Multi-Agent Learning Trajectories [7.764532811300023]
  We study inverse mechanism learning: recovering an unknown incentive-generating mechanism from observed strategic interaction traces.<n>Unlike inverse game theory and multi-agent inverse reinforcement learning, our target includes unstructured mechanism.<n>We propose DIML, a likelihood-based framework that differentiates through a model of multi-agent learning dynamics.
  arXiv  Detail & Related papers  (2026-01-25T03:49:25Z)
• PerspAct: Enhancing LLM Situated Collaboration Skills through Perspective Taking and Active Vision [2.32300953742759]
  This study evaluates whether explicitly incorporating diverse points of view using the ReAct framework can enhance an LLM's ability to understand and ground the demands of other agents.<n>We introduce active visual exploration across a suite of seven scenarios of increasing perspective-taking complexity.<n>Our results demonstrate that explicit perspective cues, combined with active exploration strategies, significantly improve the model's interpretative accuracy and collaborative effectiveness.
  arXiv  Detail & Related papers  (2025-11-11T10:54:15Z)
• Action-Dynamics Modeling and Cross-Temporal Interaction for Online Action Understanding [23.87664450145037]
  Action understanding, encompassing action detection and anticipation, plays a crucial role in numerous practical applications.<n>We propose a novel framework called the State-Specific Model (SSM), designed to unify and enhance both action detection and anticipation tasks.
  arXiv  Detail & Related papers  (2025-10-12T16:10:40Z)
• Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Control [72.00655365269]
  We present RoboMaster, a novel framework that models inter-object dynamics through a collaborative trajectory formulation.<n>Unlike prior methods that decompose objects, our core is to decompose the interaction process into three sub-stages: pre-interaction, interaction, and post-interaction.<n>Our method outperforms existing approaches, establishing new state-of-the-art performance in trajectory-controlled video generation for robotic manipulation.
  arXiv  Detail & Related papers  (2025-06-02T17:57:06Z)
• MI-DETR: An Object Detection Model with Multi-time Inquiries Mechanism [67.56918651825056]
  We propose a new decoder architecture with the parallel Multi-time Inquiries (MI) mechanism. Our MI based model, MI-DETR, outperforms all existing DETR-like models on COCO benchmark. A series of diagnostic and visualization experiments demonstrate the effectiveness, rationality, and interpretability of MI.
  arXiv  Detail & Related papers  (2025-03-03T12:19:06Z)
• Compete and Compose: Learning Independent Mechanisms for Modular World Models [57.94106862271727]
  We present COMET, a modular world model which leverages reusable, independent mechanisms across different environments. COMET is trained on multiple environments with varying dynamics via a two-step process: competition and composition. We show that COMET is able to adapt to new environments with varying numbers of objects with improved sample efficiency compared to more conventional finetuning approaches.
  arXiv  Detail & Related papers  (2024-04-23T15:03:37Z)
• LVLM-Interpret: An Interpretability Tool for Large Vision-Language Models [50.259006481656094]
  We present a novel interactive application aimed towards understanding the internal mechanisms of large vision-language models. Our interface is designed to enhance the interpretability of the image patches, which are instrumental in generating an answer. We present a case study of how our application can aid in understanding failure mechanisms in a popular large multi-modal model: LLaVA.
  arXiv  Detail & Related papers  (2024-04-03T23:57:34Z)
• Interpreting and Improving Attention From the Perspective of Large Kernel Convolution [51.06461246235176]
  We introduce Large Kernel Convolutional Attention (LKCA), a novel formulation that reinterprets attention operations as a single large- Kernel convolution. LKCA achieves competitive performance across various visual tasks, particularly in data-constrained settings.
  arXiv  Detail & Related papers  (2024-01-11T08:40:35Z)
• SlotGNN: Unsupervised Discovery of Multi-Object Representations and Visual Dynamics [15.705023986053575]
  This paper presents a novel framework for learning multi-object dynamics from visual data using unsupervised techniques. Two new architectures: SlotTransport for discovering object representations from RGB images and SlotGNN for predicting their collective dynamics from RGB images and robot interactions. With only minimal additional data, our framework robustly predicts slots and their corresponding dynamics in real-world control tasks.
  arXiv  Detail & Related papers  (2023-10-06T22:37:34Z)
• CSM-H-R: A Context Modeling Framework in Supporting Reasoning Automation for Interoperable Intelligent Systems and Privacy Protection [0.07499722271664144]
  We propose a novel framework for automation of High Level Context (HLC) reasoning across intelligent systems at scale. The design of the framework supports the sharing and inter context among intelligent systems and the components for handling CSMs and the management of hierarchy, relationship, and transition. The implementation of the framework experiments on the HLC reasoning into vector and matrix computing and presents the potential to reach next level of automation.
  arXiv  Detail & Related papers  (2023-08-21T22:21:15Z)
• MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Querying [110.83590008788745]
  Motion prediction is crucial for autonomous driving systems to understand complex driving scenarios and make informed decisions. In this paper, we propose Motion TRansformer (MTR) frameworks to address these challenges. The initial MTR framework utilizes a transformer encoder-decoder structure with learnable intention queries. We introduce an advanced MTR++ framework, extending the capability of MTR to simultaneously predict multimodal motion for multiple agents.
  arXiv  Detail & Related papers  (2023-06-30T16:23:04Z)
• A Closer Look at Reward Decomposition for High-Level Robotic Explanations [18.019811754800767]
  We propose an explainable Q-Map learning framework that combines reward decomposition with abstracted action spaces. We demonstrate the effectiveness of our framework through quantitative and qualitative analysis of two robotic scenarios.
  arXiv  Detail & Related papers  (2023-04-25T16:01:42Z)
• Pessimism meets VCG: Learning Dynamic Mechanism Design via Offline Reinforcement Learning [114.36124979578896]
  We design a dynamic mechanism using offline reinforcement learning algorithms. Our algorithm is based on the pessimism principle and only requires a mild assumption on the coverage of the offline data set.
  arXiv  Detail & Related papers  (2022-05-05T05:44:26Z)
• Structure-Regularized Attention for Deformable Object Representation [17.120035855774344]
  Capturing contextual dependencies has proven useful to improve the representational power of deep neural networks. Recent approaches that focus on modeling global context, such as self-attention and non-local operation, achieve this goal by enabling unconstrained pairwise interactions between elements. We consider learning representations for deformable objects which can benefit from context exploitation by modeling the structural dependencies that the data intrinsically possesses.
  arXiv  Detail & Related papers  (2021-06-12T03:10:17Z)
• GATSBI: Generative Agent-centric Spatio-temporal Object Interaction [9.328991021103294]
  GAT SBI is a generative model that transforms a sequence of raw observations into a structured representation. We show GAT SBI achieves superior on scene decomposition and video prediction compared to its state-of-the-art counterparts.
  arXiv  Detail & Related papers  (2021-04-09T09:45:00Z)
• Transformers with Competitive Ensembles of Independent Mechanisms [97.93090139318294]
  We propose a new Transformer layer which divides the hidden representation and parameters into multiple mechanisms, which only exchange information through attention. We study TIM on a large-scale BERT model, on the Image Transformer, and on speech enhancement and find evidence for semantically meaningful specialization as well as improved performance.
  arXiv  Detail & Related papers  (2021-02-27T21:48:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.