Representation Alignment for Just Image Transformers is not Easier than You Think

• URL: http://arxiv.org/abs/2603.14366v1
• Date: Sun, 15 Mar 2026 13:08:31 GMT
• Title: Representation Alignment for Just Image Transformers is not Easier than You Think
• Abstract summary: Representation Alignment (REPA) has emerged as a simple way to accelerate Diffusion Transformers training in latent space.<n>This paper shows that the REPA can fail for Just image Transformers (JiT)<n>We propose PixelREPA, which transforms the alignment target and constrains alignment with a Masked Transformer Adapter.
• Score: 25.669017380539064
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Representation Alignment (REPA) has emerged as a simple way to accelerate Diffusion Transformers training in latent space. At the same time, pixel-space diffusion transformers such as Just image Transformers (JiT) have attracted growing attention because they remove a dependency on a pretrained tokenizer, and then avoid the reconstruction bottleneck of latent diffusion. This paper shows that the REPA can fail for JiT. REPA yields worse FID for JiT as training proceeds and collapses diversity on image subsets that are tightly clustered in the representation space of pretrained semantic encoder on ImageNet. We trace the failure to an information asymmetry: denoising occurs in the high dimensional image space, while the semantic target is strongly compressed, making direct regression a shortcut objective. We propose PixelREPA, which transforms the alignment target and constrains alignment with a Masked Transformer Adapter that combines a shallow transformer adapter with partial token masking. PixelREPA improves both training convergence and final quality. PixelREPA reduces FID from 3.66 to 3.17 for JiT-B$/16$ and improves Inception Score (IS) from 275.1 to 284.6 on ImageNet $256 \times 256$, while achieving $> 2\times$ faster convergence. Finally, PixelREPA-H$/16$ achieves FID$=1.81$ and IS$=317.2$. Our code is available at https://github.com/kaist-cvml/PixelREPA.

Related papers

• PixelPrune: Pixel-Level Adaptive Visual Token Reduction via Predictive Coding [10.881130305325767]
  PixelPrune exploits pixel-level redundancy through predictive-coding-based compression.<n>It supports pixel-lossless compression as well as controlled lossy compression.<n>Experiments show that PixelPrune maintains competitive task accuracy while delivering up to 4.2$times$ inference speedup.
  arXiv  Detail & Related papers  (2026-04-01T13:33:27Z)
• VA-$π$: Variational Policy Alignment for Pixel-Aware Autoregressive Generation [65.22452273252428]
  VA-$$ is a post-training framework to optimize autoregressive visual generation.<n>It unifies pixel reconstruction and autoregressive modeling.<n>It reduces FID from 14.36 to 7.65 and improves IS from 86.55 to 116.70 on LlamaGen-XXL.
  arXiv  Detail & Related papers  (2025-12-22T18:54:30Z)
• PixNerd: Pixel Neural Field Diffusion [30.872185815524286]
  We propose to model the patch-wise decoding with neural field and present a single-scale, single-stage, efficient, end-to-end solution.<n>Thanks to the efficient neural field representation in PixNerd, we directly achieved 2.15 FID on ImageNet $256times256$ and 2.84 FID on ImageNet $512times512$ without any complex cascade pipeline or VAE.
  arXiv  Detail & Related papers  (2025-07-31T06:07:20Z)
• Transformer based Pluralistic Image Completion with Reduced Information Loss [72.92754600354199]
  Transformer based methods have achieved great success in image inpainting recently. They regard each pixel as a token, thus suffering from an information loss issue. We propose a new transformer based framework called "PUT"
  arXiv  Detail & Related papers  (2024-03-31T01:20:16Z)
• Accurate Image Restoration with Attention Retractable Transformer [50.05204240159985]
  We propose Attention Retractable Transformer (ART) for image restoration. ART presents both dense and sparse attention modules in the network. We conduct extensive experiments on image super-resolution, denoising, and JPEG compression artifact reduction tasks.
  arXiv  Detail & Related papers  (2022-10-04T07:35:01Z)
• Reduce Information Loss in Transformers for Pluralistic Image Inpainting [112.50657646357494]
  We propose a new transformer based framework "PUT" to keep input information as much as possible. PUT greatly outperforms state-of-the-art methods on image fidelity, especially for large masked regions and complex large-scale datasets.
  arXiv  Detail & Related papers  (2022-05-10T17:59:58Z)
• Vision Transformer with Progressive Sampling [73.60630716500154]
  We propose an iterative and progressive sampling strategy to locate discriminative regions. When trained from scratch on ImageNet, PS-ViT performs 3.8% higher than the vanilla ViT in terms of top-1 accuracy.
  arXiv  Detail & Related papers  (2021-08-03T18:04:31Z)
• Transformer-Based Deep Image Matching for Generalizable Person Re-identification [114.56752624945142]
  We investigate the possibility of applying Transformers for image matching and metric learning given pairs of images. We find that the Vision Transformer (ViT) and the vanilla Transformer with decoders are not adequate for image matching due to their lack of image-to-image attention. We propose a new simplified decoder, which drops the full attention implementation with the softmax weighting, keeping only the query-key similarity.
  arXiv  Detail & Related papers  (2021-05-30T05:38:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.