Understanding and controlling diffusion models for image generation
Abstract
Over the past five years, diffusion models have become the leading tools for generating visual content. These models learn to "denoise" corrupted (noisy) versions of clean images. They generate new samples by gradually denoising a random initial noise sample. However, steering these models to create exactly the visuals we want remains a challenge. Two questions remain only partially answered. First, what do these models actually learn? Since denoising happens across a wide range of noise levels, it is unclear which aspects of the data are modeled at each noise level. Second, how can these models be efficiently controlled? In practice, control is mostly achieved through prompt engineering, by tweaking the textual prompt, which offers limited flexibility. This thesis addresses both questions: it develops an understanding of what diffusion models learn, and uses it to design control mechanisms. The thesis presents four contributions to efficient control of text-to-image diffusion models. We first introduce VETIM, a method to learn additional vocabulary (tokens) usable in prompts. VETIM only operates on the text encoder and does not require images. However, controlling generation through text is not always sufficient. Early models such as Stable Diffusion 1 and 2 struggle to generate images in certain styles when using prompts alone. We show that the initial noise used to start the generation process imprints structure and style on the output. With Diffusion in Style, we…
Read the paper
Related studies
- Learning from First-Generation Qualitative Approaches in the IS Discipline: An Evolutionary View and Some Implications for Authors and Evaluators (PART 1/2)
- Commentary on the clinical and preclinical dosage limits of interstitially administered magnetic fluids for therapeutic hyperthermia based on current practice and efficacy models
- Prosthesis: New Technological Opportunities and Innovative Biomedical Devices
- The history and future of neural modeling for cochlear implants
- Ryanodine receptors: waking up from refractoriness
- Novel Transgenic Rabbit Model Sheds Light on the Puzzling Role of Matrix Metalloproteinase-12 in Atherosclerosis
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.