Extending 3D Capabilities of 2D Diffusion Models: Effective Controls and Scalable Techniques
Abstract
Diffusion models have made a strong impact on image and video generation, scaling to massive datasets and model sizes while enabling new applications in advertising, digital art, and content creation. However, trained primarily on 2D data without 3D priors, they excel at generating individual frames but struggle with 3D consistency, failing to maintain spatial and geometric coherence across viewpoints, which limits their use in editing and creating dynamic 3D content. This thesis presents three progressively developed approaches to improve the 3D consistency of diffusion models. First, we propose an inpainting-based approach to repurpose a 2D diffusion model for 3D novel view synthesis from a single image. We reuse pixels from the source view via monocular-depth reprojection and train the network to infill the newly revealed regions. Next, we present an approach to generate 3D-consistent multi-view images by directly incorporating explicit geometric priors into a 2D diffusion model. For this, we enforce epipolar constraints during cross-view interactions and propose 3D-aware positional encodings for the views. Finally, we combine these advances in a large diffusion transformer trained on three billion human images and refined with high-quality studio captures, capable of synthesizing 1K resolution multi-view outputs from a single image. Together, these methods demonstrate that carefully designed architectures, targeted data curation, and robust evaluation can extend…
Read the paper
Related studies
- Learning from First-Generation Qualitative Approaches in the IS Discipline: An Evolutionary View and Some Implications for Authors and Evaluators (PART 1/2)
- Commentary on the clinical and preclinical dosage limits of interstitially administered magnetic fluids for therapeutic hyperthermia based on current practice and efficacy models
- Prosthesis: New Technological Opportunities and Innovative Biomedical Devices
- The history and future of neural modeling for cochlear implants
- Ryanodine receptors: waking up from refractoriness
- Novel Transgenic Rabbit Model Sheds Light on the Puzzling Role of Matrix Metalloproteinase-12 in Atherosclerosis
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.