FreeGaussian: Guidance-free Controllable 3D Gaussian Splats with Flow Derivatives

• URL: http://arxiv.org/abs/2410.22070v1
• Date: Tue, 29 Oct 2024 14:29:21 GMT
• Title: FreeGaussian: Guidance-free Controllable 3D Gaussian Splats with Flow Derivatives
• Abstract summary: We propose FreeGaussian, that mathematically derives dynamic Gaussian motion from optical flow and camera motion. Our method enables self-supervised optimization and continuity of dynamic Gaussian motions from flow priors.
• Score: 43.087760256901234
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reconstructing controllable Gaussian splats from monocular video is a challenging task due to its inherently insufficient constraints. Widely adopted approaches supervise complex interactions with additional masks and control signal annotations, limiting their real-world applications. In this paper, we propose an annotation guidance-free method, dubbed FreeGaussian, that mathematically derives dynamic Gaussian motion from optical flow and camera motion using novel dynamic Gaussian constraints. By establishing a connection between 2D flows and 3D Gaussian dynamic control, our method enables self-supervised optimization and continuity of dynamic Gaussian motions from flow priors. Furthermore, we introduce a 3D spherical vector controlling scheme, which represents the state with a 3D Gaussian trajectory, thereby eliminating the need for complex 1D control signal calculations and simplifying controllable Gaussian modeling. Quantitative and qualitative evaluations on extensive experiments demonstrate the state-of-the-art visual performance and control capability of our method. Project page: https://freegaussian.github.io.

Related papers

• GaussianFlow SLAM: Monocular Gaussian Splatting SLAM Guided by GaussianFlow [7.598885266145037]
  We propose a monocular 3DGS-SLAM that uses optical flow as a geometry-aware cue to guide the optimization of both the scene structure and camera poses.<n>Our method achieves superior rendering quality and tracking accuracy compared with state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2026-04-17T01:34:46Z)
• Learning Explicit Continuous Motion Representation for Dynamic Gaussian Splatting from Monocular Videos [43.8174312310227]
  We present an approach for high-quality dynamic Gaussian Splatting from monocular videos.<n>We use an SE(3) B-spline motion bases with a compact set of control points.<n>Our method outperforms state-of-the-art methods in novel view synthesis.
  arXiv  Detail & Related papers  (2026-03-26T05:51:56Z)
• Scale-GS: Efficient Scalable Gaussian Splatting via Redundancy-filtering Training on Streaming Content [28.476493987089466]
  This paper presents M, a scalable Gaussian Splatting framework for efficient training in streaming tasks.<n>M achieves superior visual quality while significantly reducing training time compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-08-29T09:13:04Z)
• ParticleGS: Particle-Based Dynamics Modeling of 3D Gaussians for Prior-free Motion Extrapolation [9.59448024784555]
  We propose a novel dynamic 3D Gaussian Splatting prior-free motion extrapolation framework based on particle dynamics systems.<n>Instead of simply fitting to the observed visual frame sequence, we aim to more effectively model the gaussian particle dynamics system.<n> Experimental results demonstrate that the proposed method achieves comparable rendering quality with existing approaches in reconstruction tasks.
  arXiv  Detail & Related papers  (2025-05-26T17:46:35Z)
• GaussianFlowOcc: Sparse and Weakly Supervised Occupancy Estimation using Gaussian Splatting and Temporal Flow [0.5852077003870417]
  Occupancy estimation has become a prominent task in 3D computer vision. We present a novel approach to occupancy estimation, termed GaussianFlowOcc. It is inspired by Gaussian Splatting and replaces traditional dense voxel grids with a sparse 3D Gaussian representation.
  arXiv  Detail & Related papers  (2025-02-24T16:16:01Z)
• SplineGS: Robust Motion-Adaptive Spline for Real-Time Dynamic 3D Gaussians from Monocular Video [26.468480933928458]
  We propose SplineGS, a COLMAP-free dynamic 3D Gaussian Splatting framework for high-quality reconstruction and fast rendering from monocular videos. At its core is a novel Motion-Adaptive Spline (MAS) method, which represents continuous dynamic 3D Gaussian trajectories. We present a joint optimization strategy for camera parameter estimation and 3D Gaussian attributes, leveraging photometric and geometric consistency.
  arXiv  Detail & Related papers  (2024-12-13T09:09:14Z)
• DeSiRe-GS: 4D Street Gaussians for Static-Dynamic Decomposition and Surface Reconstruction for Urban Driving Scenes [71.61083731844282]
  We present DeSiRe-GS, a self-supervised gaussian splatting representation. It enables effective static-dynamic decomposition and high-fidelity surface reconstruction in complex driving scenarios.
  arXiv  Detail & Related papers  (2024-11-18T05:49:16Z)
• MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian Splatting [56.785233997533794]
  We propose a novel deformable 3D Gaussian splatting framework called MotionGS. MotionGS explores explicit motion priors to guide the deformation of 3D Gaussians. Experiments in the monocular dynamic scenes validate that MotionGS surpasses state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-10-10T08:19:47Z)
• GASP: Gaussian Splatting for Physic-Based Simulations [3.377471081417809]
  Physics simulation is paramount for modeling and utilizing 3D scenes in various real-world applications.<n>Existing models use additional meshing mechanisms, including triangle or tetrahedron meshing, marching cubes, or cage meshes.<n>Alternatively, we can modify the physics-grounded Newtonian dynamics to align with 3D Gaussian components.<n>Our GS for Physics-Based Simulations (GASP) pipeline uses parametrized flat Gaussian distributions.
  arXiv  Detail & Related papers  (2024-09-09T17:28:57Z)
• Dynamic Gaussian Marbles for Novel View Synthesis of Casual Monocular Videos [58.22272760132996]
  We show that existing 4D Gaussian methods dramatically fail in this setup because the monocular setting is underconstrained. We propose Dynamic Gaussian Marbles, which consist of three core modifications that target the difficulties of the monocular setting. We evaluate on the Nvidia Dynamic Scenes dataset and the DyCheck iPhone dataset, and show that Gaussian Marbles significantly outperforms other Gaussian baselines in quality.
  arXiv  Detail & Related papers  (2024-06-26T19:37:07Z)
• CoGS: Controllable Gaussian Splatting [5.909271640907126]
  Controllable Gaussian Splatting (CoGS) is a new method for capturing and re-animating 3D structures. CoGS offers real-time control of dynamic scenes without the prerequisite of pre-computing control signals. In our evaluations, CoGS consistently outperformed existing dynamic and controllable neural representations in terms of visual fidelity.
  arXiv  Detail & Related papers  (2023-12-09T20:06:29Z)
• SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes [59.23385953161328]
  Novel view synthesis for dynamic scenes is still a challenging problem in computer vision and graphics. We propose a new representation that explicitly decomposes the motion and appearance of dynamic scenes into sparse control points and dense Gaussians. Our method can enable user-controlled motion editing while retaining high-fidelity appearances.
  arXiv  Detail & Related papers  (2023-12-04T11:57:14Z)
• GAFlow: Incorporating Gaussian Attention into Optical Flow [62.646389181507764]
  We push Gaussian Attention (GA) into the optical flow models to accentuate local properties during representation learning. We introduce a novel Gaussian-Constrained Layer (GCL) which can be easily plugged into existing Transformer blocks. For reliable motion analysis, we provide a new Gaussian-Guided Attention Module (GGAM)
  arXiv  Detail & Related papers  (2023-09-28T07:46:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.