Beyond Heuristics: Learnable Density Control for 3D Gaussian Splatting
Abstract Overview
This paper introduces LeGS, a framework that replaces the handcrafted heuristic density control in 3D Gaussian Splatting (3DGS) with a learnable policy network optimized via Reinforcement Learning (RL). The policy network selects among four discrete actions—maintain, clone, split, and prune—for each Gaussian based on gradient and sensitivity features. A sensitivity-based reward function quantifies each Gaussian's marginal contribution to reconstruction quality, and the authors derive a closed-form solution that reduces the reward computation complexity from O(N²) to O(N). Experiments on Mip-NeRF 360, Tanks & Temples, and Deep Blending demonstrate that LeGS achieves the best reconstruction metrics among compared methods while maintaining compact Gaussian counts.
Novelty
The principal novelty is reformulating 3DGS density control as an RL problem with a parameterized policy network, replacing fixed heuristic densification and pruning rules. The work also introduces a sensitivity-analysis-based reward with an exact closed-form computation that avoids per-Gaussian re-rendering, and a maintain-action-based value baseline that replaces a separate critic network to stabilize PPO training.
Results
LeGS achieves the best reconstruction metrics across all three benchmark datasets compared to the evaluated baselines, improving PSNR over vanilla 3DGS by +0.81 dB on Mip-NeRF 360 and +0.99 dB on Tanks & Temples while maintaining relatively compact Gaussian counts. The closed-form sensitivity computation reduces per-step score calculation time from 30.5 ms (naive O(N²) approach) to 0.9 ms (O(N)) on the Bicycle scene, and the total training overhead compared to FastGS* is approximately 80–100 additional seconds across datasets.
Key Points
- LeGS formulates 3DGS density control as a reinforcement learning problem, replacing threshold-based heuristics with a policy network that selects per-Gaussian actions (maintain, clone, split, prune).
- A sensitivity-based reward quantifies each Gaussian's marginal contribution to reconstruction quality via an exact closed-form formulation, reducing computation complexity from O(N²) to O(N) and cutting score-calculation time from 30.5 ms to 0.9 ms in the reported efficiency test.
- Across Mip-NeRF 360, Tanks & Temples, and Deep Blending, LeGS achieves the best reconstruction quality among compared methods with compact Gaussian representations, while incurring only 80–100 seconds of additional training time relative to FastGS*.