Controllable Multi-Object Insertion for Detection Data Augmentation
Abstract Overview
This paper studies controllable multi-object insertion to augment object-detection datasets with strict category and count fidelity, annotation alignment, and source-content preservation. The authors propose SceneP2F, a two-stage framework where ScenePlan first generates scene-compatible class-labeled bounding boxes and SceneAlign-Fill renders the requested objects into those planned locations. ScenePlan jointly predicts instances under scene context and plausible-region constraints, while SceneAlign-Fill leverages geometry and style conditioning, box-token regional attention, and an annotation-region-weighted objective to synthesize objects and minimize unintended background modifications. Experiments on UAVDT, VisDrone, and BDD100K assess request fidelity and scene preservation, demonstrating downstream data-augmentation benefits for rare-class detection.
Novelty
The core novelty is the two-stage coupling of explicit scene-conditioned multi-instance layout planning with layout-aligned generative rendering for count- and category-controlled insertion. Key technical designs include query-per-instance center diffusion coupled with a separate size estimator in ScenePlan, and box-class-guided regional attention with an annotation-region-weighted flow-matching objective in SceneAlign-Fill.
Results
SceneP2F achieves the lowest request-fidelity errors across UAVDT, VisDrone, and BDD100K, outperforming baselines in mTCE on UAVDT (0.817 vs. 2.744) and BDD100K (1.924 vs. 5.523). It attains the highest DINO similarity and CLIP-I scores across all three benchmarks, along with improved outside-region PSNR. Adding SceneP2F-generated samples to UAVDT training boosts rare-class detection performance, raising bus AP50 from 22.9 to 31.0 and truck AP50 from 7.2 to 12.2.
Key Points
- SceneP2F decomposes multi-object insertion into scene-aware bounding-box layout planning (ScenePlan) and annotation-aligned image rendering (SceneAlign-Fill).
- On UAVDT, VisDrone, and BDD100K benchmarks, the method demonstrates superior request fidelity (mTCE/mCCE), visual consistency, and outside-region preservation compared to general-purpose editors and insertion baselines.
- Applying SceneP2F for targeted rare-category augmentation improves downstream YOLO11-L detection performance on long-tail classes in UAVDT.