PISCES: Annotation-free Text-to-Video Post-Training via Optimal Transport-Aligned Rewards

• URL: http://arxiv.org/abs/2602.01624v1
• Date: Mon, 02 Feb 2026 04:37:11 GMT
• Title: PISCES: Annotation-free Text-to-Video Post-Training via Optimal Transport-Aligned Rewards
• Abstract summary: We present $textttPISCES$, an annotation-free post-training algorithm that addresses limitations via a novel Dual Optimal Transport (OT)-aligned Rewards module.<n>To align reward signals with human judgment, $textttPISCES$ uses OT to bridge text and video embeddings at both distributional and discrete token levels.<n>$textttPISCES$ is the first to improve annotation-free reward supervision in generative post-training through the lens of OT.
• Score: 41.81044262983872
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Text-to-video (T2V) generation aims to synthesize videos with high visual quality and temporal consistency that are semantically aligned with input text. Reward-based post-training has emerged as a promising direction to improve the quality and semantic alignment of generated videos. However, recent methods either rely on large-scale human preference annotations or operate on misaligned embeddings from pre-trained vision-language models, leading to limited scalability or suboptimal supervision. We present $\texttt{PISCES}$, an annotation-free post-training algorithm that addresses these limitations via a novel Dual Optimal Transport (OT)-aligned Rewards module. To align reward signals with human judgment, $\texttt{PISCES}$ uses OT to bridge text and video embeddings at both distributional and discrete token levels, enabling reward supervision to fulfill two objectives: (i) a Distributional OT-aligned Quality Reward that captures overall visual quality and temporal coherence; and (ii) a Discrete Token-level OT-aligned Semantic Reward that enforces semantic, spatio-temporal correspondence between text and video tokens. To our knowledge, $\texttt{PISCES}$ is the first to improve annotation-free reward supervision in generative post-training through the lens of OT. Experiments on both short- and long-video generation show that $\texttt{PISCES}$ outperforms both annotation-based and annotation-free methods on VBench across Quality and Semantic scores, with human preference studies further validating its effectiveness. We show that the Dual OT-aligned Rewards module is compatible with multiple optimization paradigms, including direct backpropagation and reinforcement learning fine-tuning.

Related papers

• DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding [11.584150931573738]
  Text-guided Video Temporal Grounding (VTG) aims to localize the relevant segments in an untrimmed video based on text queries.<n>Recent training-free VTG approaches mitigate this issue by directly matching pretrained vision-language representations.<n>We propose DSE-VTG, a framework that addresses both frame-wise visual encoding and fixed query embeddings.
  arXiv  Detail & Related papers  (2026-09-08T15:03:41Z)
• Corpus Augmentation for Sign Language Translation via LLM-Guided Video Stitching [0.16792862237830142]
  Sign language translation holds promise for improving accessibility and enabling communication between signing and non-signing communities.<n>Large weakly-aligned datasets have enabled pre-training at scale and gloss-free methods have reduced reliance on expert annotation.<n>We propose a corpus augmentation approach that requires no additional human annotation, external sign-language video corpora, or generative video models.<n>Our augmentation, applied within the same framework, achieves +2.92 BLEU-4 without any change to architecture or training protocol.
  arXiv  Detail & Related papers  (2026-06-10T10:56:48Z)
• Contrastive Pretraining with Dual Visual Encoders for Gloss-Free Sign Language Translation [33.48154010885497]
  Sign Language Translation (SLT) aims to convert sign language videos into spoken or written text.<n>We propose a two-phase, dual visual encoder framework for gloss-free SLT.
  arXiv  Detail & Related papers  (2025-07-14T14:09:36Z)
• Free$^2$Guide: Training-Free Text-to-Video Alignment using Image LVLM [54.755023509791364]
  Free$2$Guide is a gradient-free and training-free framework for aligning generated videos with text prompts.<n>Our framework supports the flexible ensembling of multiple reward models to enhance alignment without significant computational overhead.
  arXiv  Detail & Related papers  (2024-11-26T02:14:47Z)
• Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval [31.42856682276394]
  Video Moment Retrieval (VMR) aims to retrieve temporal segments in untrimmed videos corresponding to a given language query. Existing strategies are often sub-optimal since they ignore the modality imbalance problem. We introduce Modal-Enhanced Semantic Modeling (MESM), a novel framework for more balanced alignment.
  arXiv  Detail & Related papers  (2023-12-19T13:38:48Z)
• Video-Teller: Enhancing Cross-Modal Generation with Fusion and Decoupling [79.49128866877922]
  Video-Teller is a video-language foundation model that leverages multi-modal fusion and fine-grained modality alignment. Video-Teller boosts the training efficiency by utilizing frozen pretrained vision and language modules. It capitalizes on the robust linguistic capabilities of large language models, enabling the generation of both concise and elaborate video descriptions.
  arXiv  Detail & Related papers  (2023-10-08T03:35:27Z)
• TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scale [59.01246141215051]
  We analyze the factor that leads to degradation from the perspective of language supervision. We propose a tunable-free pre-training strategy to retain the generalization ability of the text encoder. We produce a series of models, dubbed TVTSv2, with up to one billion parameters.
  arXiv  Detail & Related papers  (2023-05-23T15:44:56Z)
• Align and Prompt: Video-and-Language Pre-training with Entity Prompts [111.23364631136339]
  Video-and-language pre-training has shown promising improvements on various downstream tasks. We propose Align and Prompt: an efficient and effective video-and-language pre-training framework with better cross-modal alignment. Our code and pre-trained models will be released.
  arXiv  Detail & Related papers  (2021-12-17T15:55:53Z)
• TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment [68.08689660963468]
  A new algorithm called Token-Aware Cascade contrastive learning (TACo) improves contrastive learning using two novel techniques. We set new state-of-the-art on three public text-video retrieval benchmarks of YouCook2, MSR-VTT and ActivityNet.
  arXiv  Detail & Related papers  (2021-08-23T07:24:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.