GenMatter: Perceiving Physical Objects with Generative Matter Models

• URL: http://arxiv.org/abs/2604.22160v1
• Date: Fri, 24 Apr 2026 02:18:18 GMT
• Title: GenMatter: Perceiving Physical Objects with Generative Matter Models
• Abstract summary: Humans robustly segment moving entities that constitute independently moveable chunks of matter.<n>Existing computer vision systems lack a unified approach that works across diverse settings.<n>We propose a generative model that hierarchically groups low-level motion cues and high-level appearance features into particles.
• Score: 50.121713214509604
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human visual perception offers valuable insights for understanding computational principles of motion-based scene interpretation. Humans robustly detect and segment moving entities that constitute independently moveable chunks of matter, whether observing sparse moving dots, textured surfaces, or naturalistic scenes. In contrast, existing computer vision systems lack a unified approach that works across these diverse settings. Inspired by principles of human perception, we propose a generative model that hierarchically groups low-level motion cues and high-level appearance features into particles (small Gaussians representing local matter), and groups particles into clusters capturing coherently and independently moveable physical entities. We develop a hardware-accelerated inference algorithm based on parallelized block Gibbs sampling to recover stable particle motion and groupings. Our model operates on different kinds of inputs (random dots, stylized textures, or naturalistic RGB video), enabling it to work across settings where biological vision succeeds but existing computer vision approaches do not. We validate this unified framework across three domains: on 2D random dot kinematograms, our approach captures human object perception including graded uncertainty across ambiguous conditions; on a Gestalt-inspired dataset of camouflaged rotating objects, our approach recovers correct 3D structure from motion and thereby accurate 2D object segmentation; and on naturalistic RGB videos, our model tracks the moving 3D matter that makes up deforming objects, enabling robust object-level scene understanding. This work thus establishes a general framework for motion-based perception grounded in principles of human vision.

Related papers

• Monocular Vision Based Control Framework for Grasping [8.44254908378373]
  We present a vision-based grasping framework that targets both soft and rigid objects within a single control pipeline.<n>The proposed system combines open-vocabulary object detection, image segmentation, boundary-aware point assignment, real-time point tracking, and monocular depth estimation.<n>Results demonstrate that the framework achieves stable grasping across both soft and rigid objects using visual feedback alone.
  arXiv  Detail & Related papers  (2026-07-08T20:13:17Z)
• Compositional Motion Generation from Demonstration with Object-Centric Neural Fields [1.4916971861796382]
  We propose a generative learning-from-demonstration framework that enables compositional modeling of robotic behavior.<n>We render scenes from object-centric neural representations that integrate canonical neural fields with latent-conditioned deformations.<n>In simulation, long-horizon manipulation tasks are successfully completed using the proposed model.
  arXiv  Detail & Related papers  (2026-07-08T08:20:24Z)
• Articulated 3D Scene Graphs for Open-World Mobile Manipulation [55.97942733699124]
  We present MoMa-SG, a framework for building semantic-kinematic 3D scene graphs of articulated scenes.<n>We estimate articulation models using a novel unified twist estimation formulation.<n>We also introduce the novel Arti4D-Semantic dataset.
  arXiv  Detail & Related papers  (2026-02-18T10:40:35Z)
• PhysHMR: Learning Humanoid Control Policies from Vision for Physically Plausible Human Motion Reconstruction [52.44375492811009]
  We present PhysHMR, a unified framework that learns a visual-to-action policy for humanoid control in a physics-based simulator.<n>A key component of our approach is the pixel-as-ray strategy, which lifts 2D keypoints into 3D spatial rays and transforms them into global space.<n>PhysHMR produces high-fidelity, physically plausible motion across diverse scenarios, outperforming prior approaches in both visual accuracy and physical realism.
  arXiv  Detail & Related papers  (2025-10-02T21:01:11Z)
• TRACE: Learning 3D Gaussian Physical Dynamics from Multi-view Videos [7.616167860385134]
  We propose a new framework named TRACE to model the motion physics of complex dynamic 3D scenes.<n>By formulating each 3D point as a rigid particle with size and orientation in space, we directly learn a translation rotation dynamics system for each particle.
  arXiv  Detail & Related papers  (2025-08-13T13:43:01Z)
• Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videos [28.473117601626956]
  We develop a neural dynamics framework that combines object particles and spatial grids in a hybrid representation.<n>We demonstrate that our model learns the dynamics of diverse objects from sparse-view RGB-D recordings of robot-object interactions.<n>Our approach outperforms state-of-the-art learning-based and physics-based simulators, particularly in scenarios with limited camera views.
  arXiv  Detail & Related papers  (2025-06-18T17:59:38Z)
• Object Concepts Emerge from Motion [24.73461163778215]
  We propose a biologically inspired framework for learning object-centric visual representations in an unsupervised manner.<n>Our key insight is that motion boundary serves as a strong signal for object-level grouping.<n>Our framework is fully label-free and does not rely on camera calibration, making it scalable to large-scale unstructured video data.
  arXiv  Detail & Related papers  (2025-05-27T18:09:02Z)
• IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments [56.85804719947]
  We present IAAO, a framework that builds an explicit 3D model for intelligent agents to gain understanding of articulated objects in their environment through interaction.<n>We first build hierarchical features and label fields for each object state using 3D Gaussian Splatting (3DGS) by distilling mask features and view-consistent labels from multi-view images.<n>We then perform object- and part-level queries on the 3D Gaussian primitives to identify static and articulated elements, estimating global transformations and local articulation parameters along with affordances.
  arXiv  Detail & Related papers  (2025-04-09T12:36:48Z)
• Dynamic Scene Understanding through Object-Centric Voxelization and Neural Rendering [57.895846642868904]
  We present a 3D generative model named DynaVol-S for dynamic scenes that enables object-centric learning.<n>voxelization infers per-object occupancy probabilities at individual spatial locations.<n>Our approach integrates 2D semantic features to create 3D semantic grids, representing the scene through multiple disentangled voxel grids.
  arXiv  Detail & Related papers  (2024-07-30T15:33:58Z)
• Seeing Objects in a Cluttered World: Computational Objectness from Motion in Video [0.0]
  Perception of the visually disjoint surfaces of our world as whole objects physically distinct from those overlapping them forms the basis of our visual perception. We present a simple but novel approach to infer objectness from phenomenology without object models. We show that it delivers robust perception of individual attended objects in cluttered scenes, even with blur and camera shake.
  arXiv  Detail & Related papers  (2024-02-02T03:57:11Z)
• Object Scene Representation Transformer [56.40544849442227]
  We introduce Object Scene Representation Transformer (OSRT), a 3D-centric model in which individual object representations naturally emerge through novel view synthesis. OSRT scales to significantly more complex scenes with larger diversity of objects and backgrounds than existing methods. It is multiple orders of magnitude faster at compositional rendering thanks to its light field parametrization and the novel Slot Mixer decoder.
  arXiv  Detail & Related papers  (2022-06-14T15:40:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.