RefAlign: Representation Alignment for Reference-to-Video Generation

• URL: http://arxiv.org/abs/2603.25743v1
• Date: Thu, 26 Mar 2026 17:59:57 GMT
• Title: RefAlign: Representation Alignment for Reference-to-Video Generation
• Abstract summary: RefAlign is a representation alignment framework that aligns DiT reference-branch features to the semantic space of a visual foundation model (VFM)<n>Experiments on the OpenS2V-Eval benchmark demonstrate that RefAlign outperforms current state-of-the-art methods in TotalScore.
• Score: 53.368296137314225
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Reference-to-video (R2V) generation is a controllable video synthesis paradigm that constrains the generation process using both text prompts and reference images, enabling applications such as personalized advertising and virtual try-on. In practice, existing R2V methods typically introduce additional high-level semantic or cross-modal features alongside the VAE latent representation of the reference image and jointly feed them into the diffusion Transformer (DiT). These auxiliary representations provide semantic guidance and act as implicit alignment signals, which can partially alleviate pixel-level information leakage in the VAE latent space. However, they may still struggle to address copy--paste artifacts and multi-subject confusion caused by modality mismatch across heterogeneous encoder features. In this paper, we propose RefAlign, a representation alignment framework that explicitly aligns DiT reference-branch features to the semantic space of a visual foundation model (VFM). The core of RefAlign is a reference alignment loss that pulls the reference features and VFM features of the same subject closer to improve identity consistency, while pushing apart the corresponding features of different subjects to enhance semantic discriminability. This simple yet effective strategy is applied only during training, incurring no inference-time overhead, and achieves a better balance between text controllability and reference fidelity. Extensive experiments on the OpenS2V-Eval benchmark demonstrate that RefAlign outperforms current state-of-the-art methods in TotalScore, validating the effectiveness of explicit reference alignment for R2V tasks.

Related papers

• Bridging the Semantic-Utility Gap in Multimodal RAG via Generator-in-the-Loop Alignment [37.61076424435405]
  Vision-language models (VLMs) augmented with retrieval-augmented generation (RAG) benefit from access to external evidence.<n>Standard retrievers and rerankers optimize for semantic similarity rather than answer utility, creating a preference gap.<n>We propose a two-stage generator-in-the-loop alignment framework that closes this gap without human document-level relevance annotations.
  arXiv  Detail & Related papers  (2026-09-08T03:18:58Z)
• Representation and Reference Selection in Training-Free Synthetic Image Attribution [20.117588342501758]
  Synthetic image attribution aims at identifying the generator responsible for a given AI-generated image.<n>We provide a controlled analysis of this interaction using references and off-the-shelf pretrained representations.<n>Our results show that attribution accuracy consistently peaks at intermediate representation levels.
  arXiv  Detail & Related papers  (2026-07-13T18:10:39Z)
• Spatial-Temporal Decoupled Reference Conditioning for Identity-Preserving Text-to-Video Generation [79.94088803584262]
  Identity-preserving video generation (IPVG) aims to synthesize high-fidelity videos that follow text prompts while preserving a reference identity.<n>We propose ST-DRC, an effective Spatial-Temporal Decoupled Reference Conditioning framework for identity-preserving text-to-video generation.<n>We show that ST-DRC achieves strong identity preservation, prompt alignment, temporal consistency, and video quality with a lightweight design built on LTX-2.3.
  arXiv  Detail & Related papers  (2026-06-01T16:12:18Z)
• UniCustom: Unified Visual Conditioning for Multi-Reference Image Generation [65.53694602893042]
  We propose a unified visual conditioning framework that fuses ViT and VAE features before VLM encoding.<n>Experiments on two multi-reference generation benchmarks demonstrate that UniCustom consistently improves subject consistency, instruction following, and compositional fidelity.
  arXiv  Detail & Related papers  (2026-05-12T13:10:05Z)
• Focal Guidance: Unlocking Controllability from Semantic-Weak Layers in Video Diffusion Models [41.59364061354628]
  Image-to-Video (I2V) generation aims to synthesize a video from a reference image and a text prompt.<n>Existing I2V models prioritize visual consistency.<n>How to effectively couple this dual guidance to ensure strong adherence to the text prompt remains underexplored.
  arXiv  Detail & Related papers  (2026-01-12T07:48:26Z)
• RSRefSeg 2: Decoupling Referring Remote Sensing Image Segmentation with Foundation Models [25.265113510539546]
  Referring Remote Sensing Image provides a flexible and fine-grained framework for remote sensing scene analysis.<n>Current approaches utilize a three-stage pipeline encompassing dual-modal encoding, cross-modal interaction, and pixel decoding.<n>We propose RSRefSeg 2, a decoupling paradigm that reformulates the conventional workflow into a collaborative dual-stage framework.
  arXiv  Detail & Related papers  (2025-07-08T17:59:58Z)
• RealignDiff: Boosting Text-to-Image Diffusion Model with Coarse-to-fine Semantic Re-alignment [112.45442468794658]
  We propose a two-stage coarse-to-fine semantic re-alignment method, named RealignDiff. In the coarse semantic re-alignment phase, a novel caption reward is proposed to evaluate the semantic discrepancy between the generated image caption and the given text prompt. The fine semantic re-alignment stage employs a local dense caption generation module and a re-weighting attention modulation module to refine the previously generated images from a local semantic view.
  arXiv  Detail & Related papers  (2023-05-31T06:59:21Z)
• Reference-based Image and Video Super-Resolution via C2-Matching [100.0808130445653]
  We propose C2-Matching, which performs explicit robust matching crossing transformation and resolution. C2-Matching significantly outperforms state of the arts on the standard CUFED5 benchmark. We also extend C2-Matching to Reference-based Video Super-Resolution task, where an image taken in a similar scene serves as the HR reference image.
  arXiv  Detail & Related papers  (2022-12-19T16:15:02Z)
• Robust Reference-based Super-Resolution via C2-Matching [77.51610726936657]
  Super-Resolution (Ref-SR) has recently emerged as a promising paradigm to enhance a low-resolution (LR) input image by introducing an additional high-resolution (HR) reference image. Existing Ref-SR methods mostly rely on implicit correspondence matching to borrow HR textures from reference images to compensate for the information loss in input images. We propose C2-Matching, which produces explicit robust matching crossing transformation and resolution.
  arXiv  Detail & Related papers  (2021-06-03T16:40:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.