Subjective Camera 0.1: Bridging Human Cognition and Visual Reconstruction through Sequence-Aware Sketch-Guided Diffusion

• URL: http://arxiv.org/abs/2506.23711v2
• Date: Mon, 04 Aug 2025 16:51:20 GMT
• Title: Subjective Camera 0.1: Bridging Human Cognition and Visual Reconstruction through Sequence-Aware Sketch-Guided Diffusion
• Authors: Haoyang Chen, Dongfang Sun, Caoyuan Ma, Shiqin Wang, Kewei Zhang, Zheng Wang, Zhixiang Wang,
• Abstract summary: We introduce the concept of a subjective camera to reconstruct meaningful moments that physical cameras fail to capture.<n>We propose Subjective Camera 0.1, a framework for reconstructing real-world scenes from readily accessible subjective readouts.<n>Our approach avoids large-scale paired training data and mitigates generalization issues.
• Score: 8.477506348193
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce the concept of a subjective camera to reconstruct meaningful moments that physical cameras fail to capture. We propose Subjective Camera 0.1, a framework for reconstructing real-world scenes from readily accessible subjective readouts, i.e., textual descriptions and progressively drawn rough sketches. Built on optimization-based alignment of diffusion models, our approach avoids large-scale paired training data and mitigates generalization issues. To address the challenge of integrating multiple abstract concepts in real-world scenarios, we design a Sequence-Aware Sketch-Guided Diffusion framework with three loss terms for concept-wise sequential optimization, following the natural order of subjective readouts. Experiments on two datasets demonstrate that our method achieves state-of-the-art performance in image quality as well as spatial and semantic alignment with target scenes. User studies with 40 participants further confirm that our approach is consistently preferred.Our project page is at: subjective-camera.github.io

Related papers

• SpatialCrafter: Unleashing the Imagination of Video Diffusion Models for Scene Reconstruction from Limited Observations [44.53106180688135]
  This work takes on the challenge of reconstructing 3D scenes from sparse or single-view inputs.<n>We introduce SpatialCrafter, a framework that leverages the rich knowledge in video diffusion models to generate plausible additional observations.<n>Through a trainable camera encoder and an epipolar attention mechanism for explicit geometric constraints, we achieve precise camera control and 3D consistency.
  arXiv  Detail & Related papers  (2025-05-17T13:05:13Z)
• Uncertainty-Aware Diffusion Guided Refinement of 3D Scenes [34.19578921335553]
  Reconstructing 3D scenes from a single image is a fundamentally ill-posed task due to the severely under-constrained nature of the problem.<n>In this work, we address these inherent limitations in existing single image-to-3D scene feedforward networks.<n>To alleviate the poor performance due to insufficient information beyond the input image's view, we leverage a strong generative prior in the form of a pre-trained latent video diffusion model.
  arXiv  Detail & Related papers  (2025-03-19T23:14:27Z)
• SketchYourSeg: Mask-Free Subjective Image Segmentation via Freehand Sketches [116.1810651297801]
  SketchYourSeg establishes freehand sketches as a powerful query modality for subjective image segmentation.<n>Our evaluations demonstrate superior performance over existing approaches across diverse benchmarks.
  arXiv  Detail & Related papers  (2025-01-27T13:07:51Z)
• FreeSplatter: Pose-free Gaussian Splatting for Sparse-view 3D Reconstruction [59.77970844874235]
  We present FreeSplatter, a feed-forward reconstruction framework capable of generating high-quality 3D Gaussians from sparse-view images.<n>FreeSplatter is built upon a streamlined transformer architecture, comprising sequential self-attention blocks.<n>We show FreeSplatter's potential in enhancing the productivity of downstream applications, such as text/image-to-3D content creation.
  arXiv  Detail & Related papers  (2024-12-12T18:52:53Z)
• Diorama: Unleashing Zero-shot Single-view 3D Indoor Scene Modeling [27.577720075952225]
  We present Diorama, the first zero-shot open-world system that holistically models 3D scenes from single-view RGB observations.<n>We show the feasibility of our approach by decomposing the problem into subtasks and introduce robust, generalizable solutions to each.
  arXiv  Detail & Related papers  (2024-11-29T06:19:04Z)
• KRONC: Keypoint-based Robust Camera Optimization for 3D Car Reconstruction [58.04846444985808]
  This paper introduces KRONC, a novel approach aimed at inferring view poses by leveraging prior knowledge about the object to reconstruct and its representation through semantic keypoints. With a focus on vehicle scenes, KRONC is able to estimate the position of the views as a solution to a light optimization problem targeting the convergence of keypoints' back-projections to a singular point.
  arXiv  Detail & Related papers  (2024-09-09T08:08:05Z)
• Multi-Style Facial Sketch Synthesis through Masked Generative Modeling [17.313050611750413]
  We propose a lightweight end-to-end synthesis model that efficiently converts images to corresponding multi-stylized sketches. In this study, we overcome the issue of data insufficiency by incorporating semi-supervised learning into the training process. Our method consistently outperforms previous algorithms across multiple benchmarks.
  arXiv  Detail & Related papers  (2024-08-22T13:45:04Z)
• TALE: Training-free Cross-domain Image Composition via Adaptive Latent Manipulation and Energy-guided Optimization [59.412236435627094]
  TALE is a training-free framework harnessing the generative capabilities of text-to-image diffusion models. We equip TALE with two mechanisms dubbed Adaptive Latent Manipulation and Energy-guided Latent Optimization. Our experiments demonstrate that TALE surpasses prior baselines and attains state-of-the-art performance in image-guided composition.
  arXiv  Detail & Related papers  (2024-08-07T08:52:21Z)
• Towards Robust and Expressive Whole-body Human Pose and Shape Estimation [51.457517178632756]
  Whole-body pose and shape estimation aims to jointly predict different behaviors of the entire human body from a monocular image. Existing methods often exhibit degraded performance under the complexity of in-the-wild scenarios. We propose a novel framework to enhance the robustness of whole-body pose and shape estimation.
  arXiv  Detail & Related papers  (2023-12-14T08:17:42Z)
• Unifying Correspondence, Pose and NeRF for Pose-Free Novel View Synthesis from Stereo Pairs [57.492124844326206]
  This work delves into the task of pose-free novel view synthesis from stereo pairs, a challenging and pioneering task in 3D vision. Our innovative framework, unlike any before, seamlessly integrates 2D correspondence matching, camera pose estimation, and NeRF rendering, fostering a synergistic enhancement of these tasks.
  arXiv  Detail & Related papers  (2023-12-12T13:22:44Z)
• Learning Robust Multi-Scale Representation for Neural Radiance Fields from Unposed Images [65.41966114373373]
  We present an improved solution to the neural image-based rendering problem in computer vision. The proposed approach could synthesize a realistic image of the scene from a novel viewpoint at test time.
  arXiv  Detail & Related papers  (2023-11-08T08:18:23Z)
• FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flow [26.528667940013598]
  Reconstruction of 3D neural fields from posed images has emerged as a promising method for self-supervised representation learning. Key challenge preventing the deployment of these 3D scene learners on large-scale video data is their dependence on precise camera poses from structure-from-motion. We propose a method that jointly reconstructs camera poses and 3D neural scene representations online and in a single forward pass.
  arXiv  Detail & Related papers  (2023-05-31T20:58:46Z)
• Text-driven Visual Synthesis with Latent Diffusion Prior [37.736313030226654]
  We present a generic approach using latent diffusion models as powerful image priors for various visual synthesis tasks. We demonstrate the efficacy of our approach on three different applications, text-to-3D, StyleGAN adaptation, and layered image editing.
  arXiv  Detail & Related papers  (2023-02-16T18:59:58Z)
• Learned Spatial Representations for Few-shot Talking-Head Synthesis [68.3787368024951]
  We propose a novel approach for few-shot talking-head synthesis. We show that this disentangled representation leads to a significant improvement over previous methods.
  arXiv  Detail & Related papers  (2021-04-29T17:59:42Z)
• Deep Permutation Equivariant Structure from Motion [38.68492294795315]
  Existing deep methods produce highly accurate 3D reconstructions in stereo and multiview stereo settings. We propose a neural network architecture that recovers both the camera parameters and a scene structure by minimizing an unsupervised reprojection loss. Our experiments, conducted on a variety of datasets in both internally calibrated and uncalibrated settings, indicate that our method accurately recovers pose and structure, on par with classical state of the art methods.
  arXiv  Detail & Related papers  (2021-04-14T08:50:06Z)
• Perspective Plane Program Induction from a Single Image [85.28956922100305]
  We study the inverse graphics problem of inferring a holistic representation for natural images. We formulate this problem as jointly finding the camera pose and scene structure that best describe the input image. Our proposed framework, Perspective Plane Program Induction (P3I), combines search-based and gradient-based algorithms to efficiently solve the problem.
  arXiv  Detail & Related papers  (2020-06-25T21:18:58Z)
• Deep Self-Supervised Representation Learning for Free-Hand Sketch [51.101565480583304]
  We tackle the problem of self-supervised representation learning for free-hand sketches. Key for the success of our self-supervised learning paradigm lies with our sketch-specific designs. We show that the proposed approach outperforms the state-of-the-art unsupervised representation learning methods.
  arXiv  Detail & Related papers  (2020-02-03T16:28:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.