Generating Humanless Environment Walkthroughs from Egocentric Walking Tour Videos

• URL: http://arxiv.org/abs/2603.29036v1
• Date: Mon, 30 Mar 2026 22:08:46 GMT
• Title: Generating Humanless Environment Walkthroughs from Egocentric Walking Tour Videos
• Abstract summary: Egocentric "walking tour" videos provide a rich source of image data to develop rich and diverse visual models of environments around the world.<n>We focus on addressing this challenge by developing a generative algorithm that can realistically remove humans and their associated shadow effects from walking tour videos.<n>We show that the resulting generated clips can be used to build successful 3D/4D models of urban locations.
• Score: 24.467955018300895
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Egocentric "walking tour" videos provide a rich source of image data to develop rich and diverse visual models of environments around the world. However, the significant presence of humans in frames of these videos due to crowds and eye-level camera perspectives mitigates their usefulness in environment modeling applications. We focus on addressing this challenge by developing a generative algorithm that can realistically remove (i.e., inpaint) humans and their associated shadow effects from walking tour videos. Key to our approach is the construction of a rich semi-synthetic dataset of video clip pairs to train this generative model. Each pair in the dataset consists of an environment-only background clip, and a composite clip of walking humans with simulated shadows overlaid on the background. We randomly sourced both foreground and background components from real egocentric walking tour videos around the world to maintain visual diversity. We then used this dataset to fine-tune the state-of-the-art Casper video diffusion model for object and effects inpainting, and demonstrate that the resulting model performs far better than Casper both qualitatively and quantitatively at removing humans from walking tour clips with significant human presence and complex backgrounds. Finally, we show that the resulting generated clips can be used to build successful 3D/4D models of urban locations.

Related papers

• World Narrative Model for Highly Controllable Video Generation: A Paradigm Shift from Pixel Sampling to Physical World Orchestration [49.72473930573157]
  We introduce the World Narrative Model (WNM), a paradigm that decouples what to render -- the structured physical narrative -- from how to render.<n>WNM replaces end-to-end black-box sampling with orchestrated $4D$ pre-visualization for media generation.<n>Our human-AI platform supports automatic world generation and pre-visualization aligned with professional filmmaking pipelines.
  arXiv  Detail & Related papers  (2026-06-30T16:52:53Z)
• HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation [64.37874983401221]
  We present HumanVid, the first large-scale high-quality dataset tailored for human image animation. For the real-world data, we compile a vast collection of real-world videos from the internet. For the synthetic data, we collected 10K 3D avatar assets and leveraged existing assets of body shapes, skin textures and clothings.
  arXiv  Detail & Related papers  (2024-07-24T17:15:58Z)
• WildVidFit: Video Virtual Try-On in the Wild via Image-Based Controlled Diffusion Models [132.77237314239025]
  Video virtual try-on aims to generate realistic sequences that maintain garment identity and adapt to a person's pose and body shape in source videos. Traditional image-based methods, relying on warping and blending, struggle with complex human movements and occlusions. We reconceptualize video try-on as a process of generating videos conditioned on garment descriptions and human motion. Our solution, WildVidFit, employs image-based controlled diffusion models for a streamlined, one-stage approach.
  arXiv  Detail & Related papers  (2024-07-15T11:21:03Z)
• ActAnywhere: Subject-Aware Video Background Generation [62.57759679425924]
  Generating video background that tailors to foreground subject motion is an important problem for the movie industry and visual effects community. This task involves background that aligns with the motion and appearance of the foreground subject, while also complies with the artist's creative intention. We introduce ActAnywhere, a generative model that automates this process which traditionally requires tedious manual efforts.
  arXiv  Detail & Related papers  (2024-01-19T17:16:16Z)
• EgoEnv: Human-centric environment representations from egocentric video [60.34649902578047]
  First-person video highlights a camera-wearer's activities in the context of their persistent environment. Current video understanding approaches reason over visual features from short video clips that are detached from the underlying physical space. We present an approach that links egocentric video and the environment by learning representations that are predictive of the camera-wearer's (potentially unseen) local surroundings.
  arXiv  Detail & Related papers  (2022-07-22T22:39:57Z)
• Video Content Swapping Using GAN [1.2300363114433952]
  In this work, we will break down any frame in the video into content and pose. We first extract the pose information from a video using a pre-trained human pose detection and use a generative model to synthesize the video based on the content code and pose code.
  arXiv  Detail & Related papers  (2021-11-21T23:01:58Z)
• Image Comes Dancing with Collaborative Parsing-Flow Video Synthesis [124.48519390371636]
  Transfering human motion from a source to a target person poses great potential in computer vision and graphics applications. Previous work has either relied on crafted 3D human models or trained a separate model specifically for each target person. This work studies a more general setting, in which we aim to learn a single model to parsimoniously transfer motion from a source video to any target person.
  arXiv  Detail & Related papers  (2021-10-27T03:42:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.