PE-Field 4D: Video Generation Models as Canvas

• URL: http://arxiv.org/abs/2607.15667v1
• Date: Fri, 17 Jul 2026 06:22:14 GMT
• Title: PE-Field 4D: Video Generation Models as Canvas
• Abstract summary: We show that positional encoding in video diffusion transformers provides a useful spatial bias for geometry-aware control.<n>We introduce a geometry-aware cross-attention mechanism that enables target video latent tokens to attend to structured context tokens.<n>The resulting framework provides a simple and effective approach for controllable video generation.
• Score: 44.0217294710719
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion Transformers have recently achieved strong performance in video generation, yet controlling scene geometry under viewpoint changes and camera motion remains challenging. In this work, we revisit the role of positional encoding in video diffusion transformers and show that it provides a useful spatial bias for geometry-aware control. Specifically, if reference tokens are encoded according to their projected locations in the target view, the denoising model is encouraged to retrieve content from position aligned regions of the input video. Building on this observation, we introduce a geometry-aware cross-attention mechanism that enables target video latent tokens to attend to structured context tokens derived from reference images or frames. To establish correspondence between the reference content and the target camera trajectory, we equip the context tokens with a projected positional encoding scheme that combines target-view 2D reprojection with depth-aware disambiguation. At the same time, we preserve the original spatiotemporal positional encoding of the generated video latent, allowing geometric guidance to be injected while maintaining consistency with the video model's native latent structure. The resulting framework provides a simple and effective approach for controllable video generation. It improves spatial controllability in viewpoint-dependent editing tasks, including camera re-trajectory, novel-view video synthesis, and geometry-aware video editing, while preserving the generative prior of the underlying video diffusion model. The code is available at: https://github.com/MTLab/PE-Field.

Related papers

• RayPE: Ray-Space Positional Encoding for 3D-Aware Video Generation [43.253222907352146]
  RayPE is a positional-encoding extension that injects per-token 6D Plucker coordinates additively into the queries and keys of self-attention.<n>The resulting attention score decomposes into a content term, a geometry term, and two content and geometry cross-terms.
  arXiv  Detail & Related papers  (2026-06-25T17:51:02Z)
• Go-with-the-Track: Video Compositing and Motion Control with Point Tracking [50.22693773328114]
  Filmmaking demands precise motion control and reference image compositing.<n>We present Go-with-the-Track, which unifies both capabilities by jointly conditioning on multiple reference images and reference-anchored point-tracks.<n>Go-with-the-Track achieves superior motion and reference control in a single model.
  arXiv  Detail & Related papers  (2026-06-18T19:40:03Z)
• CRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generation [48.95338161181985]
  We propose Ray Expectation Positional Controlled (CRePE) for camera-conditioned video generation.<n>CRePE represents each image token as a depth-aware positional distribution along its source ray, capturing the projected-path geometry induced by wide-angle and fisheye cameras.<n>Design leads to more stable camera control and improves several geometry-aware and perceptual-quality metrics.
  arXiv  Detail & Related papers  (2026-05-13T03:18:26Z)
• Motion-Aware Concept Alignment for Consistent Video Editing [57.08108545219043]
  We introduce MoCA-Video (Motion-Aware Concept Alignment in Video), a training-free framework bridging the gap between image-domain semantic mixing and video.<n>Given a generated video and a user-provided reference image, MoCA-Video injects the semantic features of the reference image into a specific object within the video.<n>We evaluate MoCA's performance using the standard SSIM, image-level LPIPS, temporal LPIPS, and introduce a novel metric CASS (Conceptual Alignment Shift Score) to evaluate the consistency and effectiveness of the visual shifts between the source prompt and the modified video frames
  arXiv  Detail & Related papers  (2025-06-01T13:28:04Z)
• BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations [82.94002870060045]
  Existing video generation models struggle to follow complex text prompts and synthesize multiple objects.<n>We develop a blob-grounded video diffusion model named BlobGEN-Vid that allows users to control object motions and fine-grained object appearance.<n>We show that our framework is model-agnostic and build BlobGEN-Vid based on both U-Net and DiT-based video diffusion models.
  arXiv  Detail & Related papers  (2025-01-13T19:17:06Z)
• CoDeF: Content Deformation Fields for Temporally Consistent Video Processing [86.25225894085105]
  CoDeF is a new type of video representation consisting of a canonical content field and a temporal deformation field.<n>We experimentally show that CoDeF is able to lift image-to-image translation to video-to-video translation and lift keypoint detection to keypoint tracking without any training.
  arXiv  Detail & Related papers  (2023-08-15T17:59:56Z)
• Track Targets by Dense Spatio-Temporal Position Encoding [27.06820571703848]
  We propose a novel paradigm to encode the position of targets for target tracking in videos using transformers. The proposed encoding position provides location information to associate targets across frames beyond appearance matching. Our encoding is applied to the 2D CNN features instead of the proposed feature vectors to avoid losing positional information.
  arXiv  Detail & Related papers  (2022-10-17T22:04:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.