Resolving Representation Ambiguity in Feedforward Novel View Synthesis Transformer via Semantic-Spatial Decoupling

• URL: http://arxiv.org/abs/2605.18599v1
• Date: Mon, 18 May 2026 16:09:52 GMT
• Title: Resolving Representation Ambiguity in Feedforward Novel View Synthesis Transformer via Semantic-Spatial Decoupling
• Abstract summary: Transformer-based models have advanced feedforward novel view synthesis (NVS)<n>Current architectures mix semantic information (e.g., RGB) and spatial information (e.g., Plcker rays) into a shared feature space.<n>We propose to decouple the representation of feedforward NVS transformers into separate semantic and spatial tokens.
• Score: 117.72367737965048
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer-based models have advanced feedforward novel view synthesis (NVS). Current architectures such as GS-LRM and LVSM mix semantic information (e.g., RGB) and spatial information (e.g., Plücker rays) into a shared feature space. Since Plücker rays naturally carry lattice-like spatial structure, these designs can make the spatial bias interfere with appearance representation and degrade rendering fidelity. To this end, we propose to decouple the representation of feedforward NVS transformers into separate semantic and spatial tokens. The decoupled design keeps semantic and spatial information explicit in their branches while preserving cross-branch interaction through shared attention routing. Built on this design, we introduce optional categorized supervision and bidirectional modulation: the former provides branch-specific training signals, while the latter improves interaction between the two branches. Notably, the base decoupled design introduces virtually zero additional inference latency due to its architectural design. The proposed designs achieve consistent improvements, demonstrating effectiveness across decoder-only and encoder-decoder feedforward NVS models.

Related papers

• SSC-Priors: Exploring Semantic and Visibility Priors to Boost Lidar Semantic Scene Completion [61.665449818739155]
  This paper investigates easy strategies to boost the performance of existing networks for lidar semantic scene completion (SSC)<n>We use semantic pseudo-labels and sensor visibility information to improve existing approaches.<n>We show that endowing input point clouds with semantic pseudo-labels from off-the-shelf segmenters significantly improves the performance of existing SSC models.
  arXiv  Detail & Related papers  (2026-09-15T16:37:46Z)
• Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steering [7.62772056485722]
  We introduce emphcross-branch semantic steering, an intervention-based framework that extracts semantic directions from one branch and applies them to the other.<n>We show that steering vectors learned from the understanding branch can transfer to generation, enabling controllable image synthesis and improved semantic faithfulness.<n>Our results reveal that architectural unification does not guarantee semantic alignment, and establish cross-branch steering as a practical tool for probing multimodal representations.
  arXiv  Detail & Related papers  (2026-07-29T02:46:20Z)
• Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing [55.13802890769086]
  We present Evita, the first unified backbone specifically engineered for dedicated dense RGB-Event parsing.<n>To achieve profound modal synergy, Evita embeds a suite of intrinsic co-learning modules directly into every encoder layer.<n>Evita establishes new state-of-the-art metrics while delivering a superior accuracy-latency trade-off for real-time multimodal perception.
  arXiv  Detail & Related papers  (2026-07-10T06:52:09Z)
• Improving Relative Representations with Learned Anchors and Whitened Inner Products [0.0]
  Independently trained neural models typically converge to incompatible latent representations.<n>Relative Representations (RR) address this by mapping absolute coordinates to a shared space defined by similarities to common anchor points.<n>Traditional implementations rely on randomly sampled anchors and cosine similarity.<n>We propose a robust framework for cross-model communication based on two improvements.
  arXiv  Detail & Related papers  (2026-05-28T21:43:53Z)
• Beyond Point-Wise Matching: Structural Representation Alignment for Accelerating Diffusion Transformers [93.3976834364707]
  We propose sREPA, a structural REPresentation Alignment framework to enforce consistency in the relational geometry of feature maps.<n>By encouraging the model to internalize holistic spatial layouts and structural correlations from pre-trained features, sREPA achieves faster and more stable convergence.
  arXiv  Detail & Related papers  (2026-05-16T12:01:04Z)
• DSS-GAN: Directional State Space GAN with Mamba backbone for Class-Conditional Image Synthesis [41.99844472131922]
  We present DSS-GAN, the first generative adversarial network to employ Mamba as a hierarchical generator backbone for noise-to-image synthesis.<n>The central contribution is Directional Latent (DLR), a novel conditioning mechanism that decomposes the latent vector into direction-specific subvectors.
  arXiv  Detail & Related papers  (2026-03-18T11:58:47Z)
• From Points to Spheres: A Geometric Reinterpretation of Variational Autoencoders [0.0]
  Variational Autoencoder is typically understood from the perspective of probabilistic inference.<n>In this work, we propose a new geometric reinterpretation which complements the probabilistic view and enhances its intuitiveness.
  arXiv  Detail & Related papers  (2025-07-23T06:52:00Z)
• Cross-Layer Discrete Concept Discovery for Interpreting Language Models [13.842670153893977]
  Cross-layer VQ-VAE is a framework that uses vector quantization to map representations across layers.<n>Our approach uniquely combines top-k temperature-based sampling during quantization with EMA codebook updates.
  arXiv  Detail & Related papers  (2025-06-24T22:43:36Z)
• Cross Paradigm Representation and Alignment Transformer for Image Deraining [40.66823807648992]
  We propose a novel Cross Paradigm Representation and Alignment Transformer (CPRAformer)<n>Its core idea is the hierarchical representation and alignment, leveraging the strengths of both paradigms to aid image reconstruction.<n>We use two types of self-attention in the Transformer blocks: sparse prompt channel self-attention (SPC-SA) and spatial pixel refinement self-attention (SPR-SA)
  arXiv  Detail & Related papers  (2025-04-23T06:44:46Z)
• Redundancy-aware Transformer for Video Question Answering [71.98116071679065]
  We propose a novel transformer-based architecture, that aims to model VideoQA in a redundancy-aware manner. To address the neighboring-frame redundancy, we introduce a video encoder structure that emphasizes the object-level change in neighboring frames. As for the cross-modal redundancy, we equip our fusion module with a novel adaptive sampling, which explicitly differentiates the vision-language interactions.
  arXiv  Detail & Related papers  (2023-08-07T03:16:24Z)
• Xformer: Hybrid X-Shaped Transformer for Image Denoising [114.37510775636811]
  We present a hybrid X-shaped vision Transformer, named Xformer, which performs notably on image denoising tasks. Xformer achieves state-of-the-art performance on the synthetic and real-world image denoising tasks.
  arXiv  Detail & Related papers  (2023-03-11T16:32:09Z)
• HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning [74.76431541169342]
  Zero-shot learning (ZSL) tackles the unseen class recognition problem, transferring semantic knowledge from seen classes to unseen ones. We propose a novel hierarchical semantic-visual adaptation (HSVA) framework to align semantic and visual domains. Experiments on four benchmark datasets demonstrate HSVA achieves superior performance on both conventional and generalized ZSL.
  arXiv  Detail & Related papers  (2021-09-30T14:27:50Z)
• Do Generative Models Know Disentanglement? Contrastive Learning is All You Need [59.033559925639075]
  We propose an unsupervised and model-agnostic method: Disentanglement via Contrast (DisCo) in the Variation Space. DisCo achieves the state-of-the-art disentanglement given pretrained non-disentangled generative models, including GAN, VAE, and Flow.
  arXiv  Detail & Related papers  (2021-02-21T08:01:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.