Encoder-Decoder Manifold Alignment for Idempotent Generation
Abstract Overview
This paper studies idempotency in encoder–decoder generative models as a representation property rather than a data-space generation property. The authors argue that repeated-generation drift arises from a geometric mismatch between the manifolds learned by the encoder and decoder, so reconstruction alone does not ensure stable fixed points. They propose a model-agnostic training framework that adds a latent-space idempotency regularizer enforcing consistency between an input latent code and the code obtained after decode–re-encode, using a stop-gradient target to avoid degenerate solutions. The method is evaluated on image generation and image editing tasks, including VAEs, VQ-VAE-2, DCGAN-based settings, synthetic editing benchmarks, and a latent diffusion editing pipeline.
Novelty
The distinctive contribution is to frame idempotency as an encoder–decoder manifold alignment problem in latent space, with a projection-theoretic argument that idempotency is necessary for an optimal manifold projection. Unlike prior idempotent generation methods that enforce data-space idempotency for unconditional generation or purification, this work regularizes latent consistency to align encoder and decoder geometry for stable reconstruction and editing.
Results
Across repeated-application tests, the proposed method reduces drift and improves stability metrics over baselines and IGN-style objectives, including better FID on MNIST and CelebA and lower MSE on LFW. In editing experiments, the idempotent VAE improves MagicBrush reconstruction PSNR and SSIM, and yields better editing metrics such as DINO, LPIPS, and SSIM, while synthetic benchmarks show the strongest CAS and competitive or better edit success. The paper also notes a trade-off where the baseline slightly outperforms the aligned model on some perceptual reconstruction metrics such as LPIPS and CLIP-I.
Key Points
- The method adds a simple latent idempotency loss, E(D(E(x))) ≈ E(x), to standard encoder–decoder training in order to align encoder and decoder manifolds.
- The paper provides a theoretical claim that idempotency is necessary for the encoder–decoder composition to behave as an optimal projection, and that latent consistency is a sufficient practical condition for output-space idempotency.
- Empirical results show more stable repeated generation, improved identity/structure preservation in editing, and stronger attribute-preserving performance on synthetic editing benchmarks compared with standard baselines and IGN-style alternatives.