Twins: Learn to Predict Unified Representations with Focal Loss

• URL: http://arxiv.org/abs/2607.22531v1
• Date: Fri, 24 Jul 2026 17:59:39 GMT
• Title: Twins: Learn to Predict Unified Representations with Focal Loss
• Abstract summary: Unified multimodal models seek a shared visual token space that supports both multimodal understanding and image generation.<n>We propose Twins, a unified continuous token space formed by channel-wise concatenating ViT and VAE features on the same token grid.
• Score: 36.63304689132002
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unified multimodal models seek a shared visual token space that supports both multimodal understanding and image generation. Discrete methods unify the interface via a shared codebook, whereas continuous pipelines often rely on two disparate representations -- semantic features (e.g., ViT) for understanding and low-level latents (e.g., VAE) for synthesis -- resulting in mismatched latent spaces. We propose Twins, a unified continuous token space formed by channel-wise concatenating ViT and VAE features on the same token grid, so the sequence length is unchanged and attention cost does not increase. However, jointly modeling Twins in a Diffusion Transformer exposes a severe optimization imbalance: the model fits the ViT component well but struggles to match the VAE latent distribution. We trace this imbalance to three sources of heterogeneity: frequency bias, intrinsic dimensionality, and condition-aligned vs condition-independent uncertainty. To address it, we adapt a focal regression objective for flow matching that upweights large-error VAE dimensions, better balancing optimization across the ViT and VAE components. On ImageNet, this yields up to 10.57 gFID gain over naive MSE loss without classifier-free guidance. Twins also performs competitively on multimodal understanding benchmarks and improves reconstruction fidelity, narrowing the gap between understanding- and generation-oriented representations.

Related papers

• Where Does Generative Difficulty Reside? An Empirical Study of Target Representations [62.54876287800644]
  The target representation defines the distribution an image generator must learn, yet it is often treated as an interchangeable interface.<n>We study raw pixels, SD-VAE latents and DINOv2 as well as MAE representation-autoencoder features within a unified masked autoregressive rectified-flow model.<n>Our results show that compression, reconstruction fidelity, token dimensionality, and visible semantic clustering do not individually predict generative behavior.
  arXiv  Detail & Related papers  (2026-08-01T12:32:31Z)
• Rarity-Aware Discrete Diffusion with Spatially Consistent Decoding for Photo-Realistic Image Super-Resolution [80.28905986756685]
  DiMOO-SR is a rarity-aware multimodal discrete diffusion framework for photo-realistic image Super-Resolution.<n>Experiments on widely used real-world SR benchmarks demonstrate that DiMOO-SR achieves competitive perceptual quality with only a few parallel decoding steps.
  arXiv  Detail & Related papers  (2026-07-20T07:01:01Z)
• MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging [41.25982557850732]
  We introduce MergeTok, a unified tokenizer that jointly optimize continuous (VAE) and discrete (VQ) tokenizers within a encoder-decoder architecture.<n> MergeTok shows competitive reconstruction and generation performance on ImageNet-256, with substantially lower rFID than strong VAE and VQ models under matched token budgets.
  arXiv  Detail & Related papers  (2026-05-29T06:38:10Z)
• DUET: Dual-Paradigm Adaptive Expert Triage with Single-cell Inductive Prior for Spatial Transcriptomics Prediction [6.028508407069093]
  Inferring spatially resolved gene expression from histology images offers a cost-effective complement to spatial transcriptomics.<n>We propose DUET, a novel dual-paradigm framework that synergizes parametric prediction and memory-based retrieval under cellular inductive priors.
  arXiv  Detail & Related papers  (2026-05-13T20:43:29Z)
• V-Co: A Closer Look at Visual Representation Alignment via Co-Denoising [65.5867130156805]
  We present V-Co, a systematic study of visual co-denoising in a unified JiT-based framework.<n>Our study reveals four key ingredients for effective visual co-denoising.<n>V-Co outperforms the underlying pixel-space diffusion baseline and strong prior pixel-diffusion methods.
  arXiv  Detail & Related papers  (2026-03-17T17:01:54Z)
• UniCom: Unified Multimodal Modeling via Compressed Continuous Semantic Representations [45.861324782424326]
  Current unified multimodal models typically rely on discrete visual tokenizers to bridge the modality gap.<n>We introduce UniCom, a unified framework that harmonizes multimodal understanding and generation via compressed continuous representation.
  arXiv  Detail & Related papers  (2026-03-11T12:14:26Z)
• DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving [65.7087560656003]
  Generative diffusion models for end-to-end autonomous driving often suffer from mode collapse.<n>We propose DiffusionDriveV2, which leverages reinforcement learning to constrain low-quality modes and explore for superior trajectories.<n>This significantly enhances the overall output quality while preserving the inherent multimodality of its core Gaussian Mixture Model.
  arXiv  Detail & Related papers  (2025-12-08T17:29:52Z)
• VAEVQ: Enhancing Discrete Visual Tokenization through Variational Modeling [22.005420177236804]
  We propose VAEVQ, which comprises three key components: (1) Variational Latent Quantization (VLQ), replacing the AE with a VAE for quantization to leverage its structured and smooth latent space, thereby facilitating more effective codeword activation; (2) Representation Coherence Strategy (RCS), adaptively modulating the alignment strength between pre- and post-quantization features to enhance consistency and prevent overfitting to noise; and (3) Distribution Consistency Regularization (DCR), aligning the entire codebook distribution with the continuous latent distribution to improve utilization.
  arXiv  Detail & Related papers  (2025-11-10T09:07:23Z)
• Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials [34.77694214755808]
  We introduce Visual-Contrast Attention (VCA), a drop-in replacement for Multi-Head Self-Attention (MHSA)<n>VCA injects an explicit notion of discrimination while reducing the theoretical complexity from O(N N C) to O(N n C) with n N.<n>The module adds fewer than 0.3M parameters to a DeiT-Tiny backbone, requires no extra FLOPs, and is wholly architecture-agnostic.
  arXiv  Detail & Related papers  (2025-11-02T07:04:12Z)
• Can We Achieve Efficient Diffusion without Self-Attention? Distilling Self-Attention into Convolutions [94.21989689001848]
  We propose (Delta)ConvFusion to replace conventional self-attention modules with Pyramid Convolution Blocks ((Delta)ConvBlocks)<n>By distilling attention patterns into localized convolutional operations while keeping other components frozen, (Delta)ConvFusion achieves performance comparable to transformer-based counterparts while reducing computational cost by 6929$times$ and surpassing LinFusion by 5.42$times$ in efficiency--all without compromising generative fidelity.
  arXiv  Detail & Related papers  (2025-04-30T03:57:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.