Improving Viewpoint Consistency in 3D Generation via Structure Feature and CLIP Guidance

• URL: http://arxiv.org/abs/2412.02287v4
• Date: Thu, 14 Aug 2025 00:28:36 GMT
• Title: Improving Viewpoint Consistency in 3D Generation via Structure Feature and CLIP Guidance
• Abstract summary: Current methods often suffer from geometric inconsistencies, commonly referred to as the Janus Problem.<n>This paper identifies the root cause of the Janus Problem: viewpoint generation bias in diffusion models.<n>We propose a tuning-free approach called the Attention and CLIP Guidance (ACG) mechanism.
• Score: 17.867010229092735
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite recent advances in text-to-3D generation techniques, current methods often suffer from geometric inconsistencies, commonly referred to as the Janus Problem. This paper identifies the root cause of the Janus Problem: viewpoint generation bias in diffusion models, which creates a significant gap between the actual generated viewpoint and the expected one required for optimizing the 3D model. To address this issue, we propose a tuning-free approach called the Attention and CLIP Guidance (ACG) mechanism. ACG enhances desired viewpoints by adaptively controlling cross-attention maps, employs CLIP-based view-text similarities to filter out erroneous viewpoints, and uses a coarse-to-fine optimization strategy with staged prompts to progressively refine 3D generation. Extensive experiments demonstrate that our method significantly reduces the Janus Problem without compromising generation speed, establishing ACG as an efficient, plug-and-play component for existing text-to-3D frameworks.

Related papers

• CGGS: Consistency-Augmented Geometric Gaussian Splatting for Ego-centric 3D Scene Generation [14.481547285312928]
  CGGS aims to enhance 3D-content-awareness and address geometric distortions in ego-centric scene generation.<n>Ego-centric Generator is proposed by fine-tuning a Multi-View Latent Diffusion Model with consistency-augmented loss.<n> layout Decorator leverages optical flow and point-track correspondence to estimate depth.<n> Geometric Refiner is proposed to enhance 3D Gaussian reconstruction via an entropy-based Mutual Information Depth Loss.
  arXiv  Detail & Related papers  (2026-07-04T11:06:41Z)
• Structural Energy Guidance for View-Consistent Text-to-3D Generation [18.973527029488746]
  This work identifies viewpoint bias in 2D diffusion priors as the main cause and proposes Structural Energy-Guided Sampling (SEGS) to improve multi-view consistency.<n>SEGS constructs a structural energy in the PCA subspace of U-Net features and injects its gradient into the denoising process.<n> Experiments show that SEGS reduces the Janus Rate by about 10% on average and improves View-CS scores across multiple baselines, including DreamFusion, Magic3D, and LucidDreamer.
  arXiv  Detail & Related papers  (2026-05-19T14:08:52Z)
• ERGO: Excess-Risk-Guided Optimization for High-Fidelity Monocular 3D Gaussian Splatting [63.138778159026934]
  We propose an adaptive optimization framework guided by excess risk decomposition, termed ERGO.<n> ERGO dynamically estimates the view-specific excess risk and adaptively adjust loss weights during optimization.<n>Experiments on the Google Scanned Objects dataset and the OmniObject3D dataset demonstrate the superiority of ERGO over existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-02-10T20:44:43Z)
• LSS3D: Learnable Spatial Shifting for Consistent and High-Quality 3D Generation from Single-Image [12.683334603689502]
  We propose a high-quality image-to-3D approach, named LSS3D, with learnable spatial shifting to handle the multiview inconsistencies and non-frontal input view.<n>Specifically, we assign learnable spatial shifting parameters to each view, and adjust each view towards a spatially consistent target, guided by the reconstructed mesh.<n>Our method consistently achieves leading results in both geometric and texture evaluation metrics across more flexible input viewpoints.
  arXiv  Detail & Related papers  (2025-11-15T13:14:25Z)
• OracleGS: Grounding Generative Priors for Sparse-View Gaussian Splatting [78.70702961852119]
  OracleGS reconciles generative completeness with regressive fidelity for sparse view Gaussian Splatting.<n>Our approach conditions the powerful generative prior on multi-view geometric evidence, filtering hallucinatory artifacts while preserving plausible completions in under-constrained regions.
  arXiv  Detail & Related papers  (2025-09-27T11:19:32Z)
• FastVGGT: Training-Free Acceleration of Visual Geometry Transformer [83.67766078575782]
  VGGT is a state-of-the-art feed-forward visual geometry model.<n>We propose FastVGGT, which leverages token merging in the 3D domain through a training-free mechanism for accelerating VGGT.<n>With 1000 input images, FastVGGT achieves a 4x speedup over VGGT while mitigating error accumulation in long-sequence scenarios.
  arXiv  Detail & Related papers  (2025-09-02T17:54:21Z)
• Structural Energy-Guided Sampling for View-Consistent Text-to-3D [18.973527029488746]
  Text-to-3D generation often suffers from the Janus problem, where objects collapse into duplicated or distorted geometry from other angles.<n>We propose Structural Energy-Guided Sampling (SEGS), a training-free, plug-and-play framework that enforces multi-view consistency entirely at sampling time.
  arXiv  Detail & Related papers  (2025-08-23T06:26:04Z)
• Intern-GS: Vision Model Guided Sparse-View 3D Gaussian Splatting [95.61137026932062]
  Intern-GS is a novel approach to enhance the process of sparse-view Gaussian splatting.<n>We show that Intern-GS achieves state-of-the-art rendering quality across diverse datasets.
  arXiv  Detail & Related papers  (2025-05-27T05:17:49Z)
• Taming Video Diffusion Prior with Scene-Grounding Guidance for 3D Gaussian Splatting from Sparse Inputs [28.381287866505637]
  We propose a reconstruction by generation pipeline that leverages learned priors from video diffusion models to provide plausible interpretations for regions outside the field of view or occluded. We introduce a novel scene-grounding guidance based on rendered sequences from an optimized 3DGS, which tames the diffusion model to generate consistent sequences. Our method significantly improves upon the baseline and achieves state-of-the-art performance on challenging benchmarks.
  arXiv  Detail & Related papers  (2025-03-07T01:59:05Z)
• RIGI: Rectifying Image-to-3D Generation Inconsistency via Uncertainty-aware Learning [27.4552892119823]
  inconsistencies in multi-view snapshots frequently introduce noise and artifacts along object boundaries, undermining the 3D reconstruction process.<n>We leverage 3D Gaussian Splatting (3DGS) for 3D reconstruction, and explicitly integrate uncertainty-aware learning into the reconstruction process.<n>We apply adaptive pixel-wise loss weighting to regularize the models, reducing reconstruction intensity in high-uncertainty regions.
  arXiv  Detail & Related papers  (2024-11-28T02:19:28Z)
• MVGaussian: High-Fidelity text-to-3D Content Generation with Multi-View Guidance and Surface Densification [13.872254142378772]
  This paper introduces a unified framework for text-to-3D content generation. Our approach utilizes multi-view guidance to iteratively form the structure of the 3D model. We also introduce a novel densification algorithm that aligns gaussians close to the surface.
  arXiv  Detail & Related papers  (2024-09-10T16:16:34Z)
• Geometry-Aware Score Distillation via 3D Consistent Noising and Gradient Consistency Modeling [31.945761751215134]
  We introduce 3D consistent noising, geometry-based gradient warping and novel gradient consistency loss. We successfully address the geometric inconsistency problems in text-to-3D generation task with minimal cost and being compatible with existing score distillation-based models.
  arXiv  Detail & Related papers  (2024-06-24T14:58:17Z)
• VividDreamer: Towards High-Fidelity and Efficient Text-to-3D Generation [69.68568248073747]
  We propose Pose-dependent Consistency Distillation Sampling (PCDS), a novel yet efficient objective for diffusion-based 3D generation tasks. PCDS builds the pose-dependent consistency function within diffusion trajectories, allowing to approximate true gradients through minimal sampling steps. For efficient generation, we propose a coarse-to-fine optimization strategy, which first utilizes 1-step PCDS to create the basic structure of 3D objects, and then gradually increases PCDS steps to generate fine-grained details.
  arXiv  Detail & Related papers  (2024-06-21T08:21:52Z)
• CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models [52.29804282879437]
  CFG++ is a novel approach that tackles the offmanifold challenges inherent to traditional CFG. It offers better inversion-to-image generation, invertibility, smaller guidance scales, reduced mode collapse, etc. It can be easily integrated into high-order diffusion solvers and naturally extends to distilled diffusion models.
  arXiv  Detail & Related papers  (2024-06-12T10:40:10Z)
• GECO: Generative Image-to-3D within a SECOnd [51.20830808525894]
  We introduce GECO, a novel method for high-quality 3D generative modeling that operates within a second. GECO achieves high-quality image-to-3D mesh generation with an unprecedented level of efficiency.
  arXiv  Detail & Related papers  (2024-05-30T17:58:00Z)
• 3D Face Modeling via Weakly-supervised Disentanglement Network joint Identity-consistency Prior [62.80458034704989]
  Generative 3D face models featuring disentangled controlling factors hold immense potential for diverse applications in computer vision and computer graphics. Previous 3D face modeling methods face a challenge as they demand specific labels to effectively disentangle these factors. This paper introduces a Weakly-Supervised Disentanglement Framework, denoted as WSDF, to facilitate the training of controllable 3D face models without an overly stringent labeling requirement.
  arXiv  Detail & Related papers  (2024-04-25T11:50:47Z)
• Learn to Optimize Denoising Scores for 3D Generation: A Unified and Improved Diffusion Prior on NeRF and 3D Gaussian Splatting [60.393072253444934]
  We propose a unified framework aimed at enhancing the diffusion priors for 3D generation tasks. We identify a divergence between the diffusion priors and the training procedures of diffusion models that substantially impairs the quality of 3D generation.
  arXiv  Detail & Related papers  (2023-12-08T03:55:34Z)
• LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching [33.696757740830506]
  Recent advancements in text-to-3D generation have shown promise. Many methods base themselves on Score Distillation Sampling (SDS) We propose Interval Score Matching (ISM) to counteract over-smoothing.
  arXiv  Detail & Related papers  (2023-11-19T09:59:09Z)
• Instant3D: Instant Text-to-3D Generation [101.25562463919795]
  We propose a novel framework for fast text-to-3D generation, dubbed Instant3D. Instant3D is able to create a 3D object for an unseen text prompt in less than one second with a single run of a feedforward network.
  arXiv  Detail & Related papers  (2023-11-14T18:59:59Z)
• EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior [59.25950280610409]
  We propose a robust high-quality 3D content generation pipeline by exploiting orthogonal-view image guidance. In this paper, we introduce a novel 2D diffusion model that generates an image consisting of four sub-images based on the given text prompt. We also present a 3D synthesis network that can further improve the details of the generated 3D contents.
  arXiv  Detail & Related papers  (2023-08-25T07:39:26Z)
• Debiasing Scores and Prompts of 2D Diffusion for View-consistent Text-to-3D Generation [38.032010026146146]
  We propose two approaches to debias the score-distillation frameworks for view-consistent text-to-3D generation. One of the most notable issues is the Janus problem, where the most canonical view of an object appears in other views. Our methods improve the realism of the generated 3D objects by significantly reducing artifacts and achieve a good trade-off between faithfulness to the 2D diffusion models and 3D consistency with little overhead.
  arXiv  Detail & Related papers  (2023-03-27T17:31:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.