Rendering Multi-Human and Multi-Object with 3D Gaussian Splatting

• URL: http://arxiv.org/abs/2604.02996v2
• Date: Wed, 08 Apr 2026 01:47:29 GMT
• Title: Rendering Multi-Human and Multi-Object with 3D Gaussian Splatting
• Abstract summary: Reconstructing dynamic scenes with multiple interacting humans and objects from sparse-view inputs is critical for creating high-fidelity digital twins for robotics and VR/AR.<n>This problem, which we term Multi-Human Multi-Object rendering, presents two significant obstacles.<n>We propose MM-GS, a novel hierarchical framework built upon 3D Splatting.
• Score: 47.35917581534084
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reconstructing dynamic scenes with multiple interacting humans and objects from sparse-view inputs is a critical yet challenging task, essential for creating high-fidelity digital twins for robotics and VR/AR. This problem, which we term Multi-Human Multi-Object (MHMO) rendering, presents two significant obstacles: achieving view-consistent representations for individual instances under severe mutual occlusion, and explicitly modeling the complex and combinatorial dependencies that arise from their interactions. To overcome these challenges, we propose MM-GS, a novel hierarchical framework built upon 3D Gaussian Splatting. Our method first employs a Per-Instance Multi-View Fusion module to establish a robust and consistent representation for each instance by aggregating visual information across all available views. Subsequently, a Scene-Level Instance Interaction module operates on a global scene graph to reason about relationships between all participants, refining their attributes to capture subtle interaction effects. Extensive experiments on challenging datasets demonstrate that our method significantly outperforms strong baselines, producing state-of-the-art results with high-fidelity details and plausible inter-instance contacts.

Related papers

• Real-World Cooperative Bimanual Dexterous Grasp of Large Objects from Single-View Observations [18.70687400045831]
  Bimanual dexterous grasping of large objects is a critical challenge in robotic manipulation.<n>We propose a real-world bimanual grasping framework that includes a multimodal dataset capturing joint angles, visual observations and force signals.<n>Our approach enables the synthesis of executable bimanual grasps from single-view inputs, reducing dependence on complete 3D object models.
  arXiv  Detail & Related papers  (2026-08-11T02:24:27Z)
• MOCHI: Motion Enhancement of Collaborative Human-object Interactions [9.360852329235477]
  Collaborative human-object interaction shows and complex movements that require mutual anticipation and continuous adjustment between participants and the shared object.<n>We present MOCHI, a two-stage framework for enhancing noisy MHOI data.<n>Our approach first generates physically plausible hand grasps through optimization from noisy body input, producing grasps that are both physically plausible and semantically consistent with the body pose.<n>During the optimization process, we introduce optimization objectives to encode human-object and human-human interaction information within these single-person priors.
  arXiv  Detail & Related papers  (2026-06-16T17:58:44Z)
• TacUMI: A Multi-Modal Universal Manipulation Interface for Contact-Rich Tasks [35.05859151174601]
  We build on the idea of the handheld demonstration device Universal Manipulation Interface (UMI)<n>We introduce TacUMI, a multi-modal data collection system that integrates ViTac sensors, force-torque sensor, and pose tracker into a compact, robot-compatible gripper design.<n>We then propose a multi-modal segmentation framework that leverages temporal models to detect semantically meaningful event boundaries.
  arXiv  Detail & Related papers  (2026-01-21T00:14:28Z)
• Asset-Driven Sematic Reconstruction of Dynamic Scene with Multi-Human-Object Interactions [41.29588736908775]
  3D geometry modeling of dynamic scenes is crucial for applications like AR/VR, gaming, and embodied AI.<n>We propose a hybrid approach that combines the advantages of 1) 3D generative models for generating high-fidelity meshes of the scene elements, 2) Semantic-aware deformation, and 3) GS-based optimization of the individual elements.<n>Our method outperforms the state-of-the-art method in producing better surface reconstruction of such scenes.
  arXiv  Detail & Related papers  (2025-11-29T16:36:22Z)
• DIPO: Dual-State Images Controlled Articulated Object Generation Powered by Diverse Data [67.99373622902827]
  DIPO is a framework for controllable generation of articulated 3D objects from a pair of images.<n>We propose a dual-image diffusion model that captures relationships between the image pair to generate part layouts and joint parameters.<n>We propose PM-X, a large-scale dataset of complex articulated 3D objects, accompanied by rendered images, URDF annotations, and textual descriptions.
  arXiv  Detail & Related papers  (2025-05-26T18:55:14Z)
• IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments [56.85804719947]
  We present IAAO, a framework that builds an explicit 3D model for intelligent agents to gain understanding of articulated objects in their environment through interaction.<n>We first build hierarchical features and label fields for each object state using 3D Gaussian Splatting (3DGS) by distilling mask features and view-consistent labels from multi-view images.<n>We then perform object- and part-level queries on the 3D Gaussian primitives to identify static and articulated elements, estimating global transformations and local articulation parameters along with affordances.
  arXiv  Detail & Related papers  (2025-04-09T12:36:48Z)
• SM3Det: A Unified Model for Multi-Modal Remote Sensing Object Detection [79.58755811919366]
  This paper introduces a new task called Multi-Modal datasets and Multi-Task Object Detection (M2Det) for remote sensing.<n>It is designed to accurately detect horizontal or oriented objects from any sensor modality.<n>This task poses challenges due to 1) the trade-offs involved in managing multi-modal modelling and 2) the complexities of multi-task optimization.
  arXiv  Detail & Related papers  (2024-12-30T02:47:51Z)
• SM$^3$: Self-Supervised Multi-task Modeling with Multi-view 2D Images for Articulated Objects [24.737865259695006]
  We propose a self-supervised interaction perception method, referred to as SM$3$, to model articulated objects. By constructing 3D geometries and textures from the captured 2D images, SM$3$ achieves integrated optimization of movable part and joint parameters. Evaluations demonstrate that SM$3$ surpasses existing benchmarks across various categories and objects, while its adaptability in real-world scenarios has been thoroughly validated.
  arXiv  Detail & Related papers  (2024-01-17T11:15:09Z)
• Multi-Grained Multimodal Interaction Network for Entity Linking [65.30260033700338]
  Multimodal entity linking task aims at resolving ambiguous mentions to a multimodal knowledge graph. We propose a novel Multi-GraIned Multimodal InteraCtion Network $textbf(MIMIC)$ framework for solving the MEL task.
  arXiv  Detail & Related papers  (2023-07-19T02:11:19Z)
• MetaGraspNet: A Large-Scale Benchmark Dataset for Scene-Aware Ambidextrous Bin Picking via Physics-based Metaverse Synthesis [72.85526892440251]
  We introduce MetaGraspNet, a large-scale photo-realistic bin picking dataset constructed via physics-based metaverse synthesis. The proposed dataset contains 217k RGBD images across 82 different article types, with full annotations for object detection, amodal perception, keypoint detection, manipulation order and ambidextrous grasp labels for a parallel-jaw and vacuum gripper. We also provide a real dataset consisting of over 2.3k fully annotated high-quality RGBD images, divided into 5 levels of difficulties and an unseen object set to evaluate different object and layout properties.
  arXiv  Detail & Related papers  (2022-08-08T08:15:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.