S^2Former-OR: Single-Stage Bimodal Transformer for Scene Graph
Generation in OR
- URL: http://arxiv.org/abs/2402.14461v1
- Date: Thu, 22 Feb 2024 11:40:49 GMT
- Title: S^2Former-OR: Single-Stage Bimodal Transformer for Scene Graph
Generation in OR
- Authors: Jialun Pei, Diandian Guo, Jingyang Zhang, Manxi Lin, Yueming Jin,
Pheng-Ann Heng
- Abstract summary: Scene graph generation (SGG) of surgical procedures is crucial in enhancing holistically cognitive intelligence in the operating room (OR)
Previous works have primarily relied on the multi-stage learning that generates semantic scene graphs dependent on intermediate processes with pose estimation and object detection.
In this study, we introduce a novel single-stage bimodal transformer framework for SGG in the OR, termed S2Former-OR.
- Score: 52.964721233679406
- License: http://creativecommons.org/publicdomain/zero/1.0/
- Abstract: Scene graph generation (SGG) of surgical procedures is crucial in enhancing
holistically cognitive intelligence in the operating room (OR). However,
previous works have primarily relied on the multi-stage learning that generates
semantic scene graphs dependent on intermediate processes with pose estimation
and object detection, which may compromise model efficiency and efficacy, also
impose extra annotation burden. In this study, we introduce a novel
single-stage bimodal transformer framework for SGG in the OR, termed
S^2Former-OR, aimed to complementally leverage multi-view 2D scenes and 3D
point clouds for SGG in an end-to-end manner. Concretely, our model embraces a
View-Sync Transfusion scheme to encourage multi-view visual information
interaction. Concurrently, a Geometry-Visual Cohesion operation is designed to
integrate the synergic 2D semantic features into 3D point cloud features.
Moreover, based on the augmented feature, we propose a novel relation-sensitive
transformer decoder that embeds dynamic entity-pair queries and relational
trait priors, which enables the direct prediction of entity-pair relations for
graph generation without intermediate steps. Extensive experiments have
validated the superior SGG performance and lower computational cost of
S^2Former-OR on 4D-OR benchmark, compared with current OR-SGG methods, e.g., 3%
Precision increase and 24.2M reduction in model parameters. We further compared
our method with generic single-stage SGG methods with broader metrics for a
comprehensive evaluation, with consistently better performance achieved. The
code will be made available.
Related papers
- Graph and Skipped Transformer: Exploiting Spatial and Temporal Modeling Capacities for Efficient 3D Human Pose Estimation [36.93661496405653]
We take a global approach to exploit Transformer-temporal information with a concise Graph and Skipped Transformer architecture.
Specifically, in 3D pose stage, coarse-grained body parts are deployed to construct a fully data-driven adaptive model.
Experiments are conducted on Human3.6M, MPI-INF-3DHP and Human-Eva benchmarks.
arXiv Detail & Related papers (2024-07-03T10:42:09Z) - RT-GS2: Real-Time Generalizable Semantic Segmentation for 3D Gaussian Representations of Radiance Fields [6.071025178912125]
We introduce RT-GS2, the first generalizable semantic segmentation method employing Gaussian Splatting.
Our method achieves real-time performance of 27.03 FPS, marking an astonishing 901 times speedup compared to existing approaches.
arXiv Detail & Related papers (2024-05-28T10:34:28Z) - CLIP-GS: CLIP-Informed Gaussian Splatting for Real-time and View-consistent 3D Semantic Understanding [32.76277160013881]
We present CLIP-GS, which integrates semantics from Contrastive Language-Image Pre-Training (CLIP) into Gaussian Splatting.
SAC exploits the inherent unified semantics within objects to learn compact yet effective semantic representations of 3D Gaussians.
We also introduce a 3D Coherent Self-training (3DCS) strategy, resorting to the multi-view consistency originated from the 3D model.
arXiv Detail & Related papers (2024-04-22T15:01:32Z) - Motion-aware 3D Gaussian Splatting for Efficient Dynamic Scene Reconstruction [89.53963284958037]
We propose a novel motion-aware enhancement framework for dynamic scene reconstruction.
Specifically, we first establish a correspondence between 3D Gaussian movements and pixel-level flow.
For the prevalent deformation-based paradigm that presents a harder optimization problem, a transient-aware deformation auxiliary module is proposed.
arXiv Detail & Related papers (2024-03-18T03:46:26Z) - SGTR+: End-to-end Scene Graph Generation with Transformer [42.396971149458324]
Scene Graph Generation (SGG) remains a challenging visual understanding task due to its compositional property.
Most previous works adopt a bottom-up, two-stage or point-based, one-stage approach, which often suffers from high time complexity or suboptimal designs.
We propose a novel SGG method to address the aforementioned issues, formulating the task as a bipartite graph construction problem.
arXiv Detail & Related papers (2024-01-23T15:18:20Z) - Towards a Unified Transformer-based Framework for Scene Graph Generation
and Human-object Interaction Detection [116.21529970404653]
We introduce SG2HOI+, a unified one-step model based on the Transformer architecture.
Our approach employs two interactive hierarchical Transformers to seamlessly unify the tasks of SGG and HOI detection.
Our approach achieves competitive performance when compared to state-of-the-art HOI methods.
arXiv Detail & Related papers (2023-11-03T07:25:57Z) - Single-Stage Visual Relationship Learning using Conditional Queries [60.90880759475021]
TraCQ is a new formulation for scene graph generation that avoids the multi-task learning problem and the entity pair distribution.
We employ a DETR-based encoder-decoder conditional queries to significantly reduce the entity label space as well.
Experimental results show that TraCQ not only outperforms existing single-stage scene graph generation methods, it also beats many state-of-the-art two-stage methods on the Visual Genome dataset.
arXiv Detail & Related papers (2023-06-09T06:02:01Z) - Multi-body SE(3) Equivariance for Unsupervised Rigid Segmentation and
Motion Estimation [49.56131393810713]
We present an SE(3) equivariant architecture and a training strategy to tackle this task in an unsupervised manner.
Our method excels in both model performance and computational efficiency, with only 0.25M parameters and 0.92G FLOPs.
arXiv Detail & Related papers (2023-06-08T22:55:32Z) - Similarity-Aware Fusion Network for 3D Semantic Segmentation [87.51314162700315]
We propose a similarity-aware fusion network (SAFNet) to adaptively fuse 2D images and 3D point clouds for 3D semantic segmentation.
We employ a late fusion strategy where we first learn the geometric and contextual similarities between the input and back-projected (from 2D pixels) point clouds.
We show that SAFNet significantly outperforms existing state-of-the-art fusion-based approaches across various data integrity.
arXiv Detail & Related papers (2021-07-04T09:28:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.