GraphWrit3R: End-to-End 3D Scene Graph Writing
Abstract Overview
GraphWrit3R is an end-to-end method for generating 3D scene graphs directly from a point cloud, Gaussian splats, or both, outputting a structured JSON graph of objects, attributes, and relationships. The system encodes point clouds with Sonata and Gaussian splats with Chorus on a shared voxel grid, fuses co-located tokens, and decodes the scene graph with a fine-tuned language model. To support a single model across input modalities, the paper introduces a per-voxel contrastive alignment loss that brings the two encoder feature spaces into correspondence. Experiments on 3DSSG and related evaluations show strong scene-graph recall, analysis of modality and fusion choices, and evidence that joint relational supervision can also improve object detection.
Novelty
The paper frames 3D scene graph generation as direct scene graph writing: a fully end-to-end latent pipeline that emits structured JSON without explicit intermediate object pipelines, proprietary APIs, or ground-truth object annotations at inference. It also introduces a unified multi-modal design for point clouds and Gaussian splats, coupled with an asymmetric per-voxel contrastive alignment loss so one set of weights can handle different 3D inputs.
Results
On 3DSSG and RIO10 evaluations, GraphWrit3R achieves the highest overall recall across object, predicate, and triplet metrics among compared methods without requiring ground-truth object boxes at inference. It demonstrates strong zero-shot transfer to ScanNet, especially on predicate recall, and joint relational supervision improves ScanNet20 object detection over SpatialLM. Ablations show that point-cloud inference is strongest and that asymmetric contrastive alignment with simple average fusion is crucial for multi-modal performance.
Key Points
- GraphWrit3R directly decodes a full 3D scene graph as JSON from 3D inputs, jointly predicting objects and relations in a single end-to-end latent model.
- The method supports point clouds, Gaussian splats, or both by aligning Sonata and Chorus voxel features with an asymmetric contrastive loss before LLM decoding.
- Empirically, the approach achieves state-of-the-art scene graph recall on the reported benchmarks and demonstrates that relational supervision benefits object detection.