Text as Controller and Crutch: Generative Control and Over-reliance in Vision-Language Models
Abstract
Text has become a central interface in modern vision-language models. In visual generative models, text provides an intuitive mechanism for controlling image and video synthesis. In multimodal large language models, however, text can also become a shortcut that causes models to underutilize visual information. This dissertation studies these two roles of text: as a controller for visual generation, and as a crutch that reveals limitations in multimodal understanding. The first part of this dissertation investigates text-based controllability in visual generation. I study how natural-language instructions can guide image editing, image generation, and video synthesis across different forms of visual control. For image editing, I develop a multimodal large language model agent that translates user requests into executable editing operations in professional software. Since ground-truth edits are difficult to define, I further introduce a generalist reward model that evaluates editing results through multimodal reasoning and provides scalar feedback for reinforcement learning. For video generation, I study text-guided camera control, where natural-language camera descriptions specify complex camera motions. By synthesizing controllable training data and disentangling visual appearance from camera motion, this work enables more flexible video generation guided by both content and camera instructions. I also examine diffusion-based image generation and editing, showing how…
Read the paper
Related studies
- Learning from First-Generation Qualitative Approaches in the IS Discipline: An Evolutionary View and Some Implications for Authors and Evaluators (PART 1/2)
- Commentary on the clinical and preclinical dosage limits of interstitially administered magnetic fluids for therapeutic hyperthermia based on current practice and efficacy models
- Prosthesis: New Technological Opportunities and Innovative Biomedical Devices
- The history and future of neural modeling for cochlear implants
- Ryanodine receptors: waking up from refractoriness
- Novel Transgenic Rabbit Model Sheds Light on the Puzzling Role of Matrix Metalloproteinase-12 in Atherosclerosis
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.