Architectural Framework for Multi-Modal Video Generation via Fine-Tuned Stable Diffusion Models
Abstract
In recent years, large language models (LLMs) like ChatGPT have driven advancements in artificial intelligence (AI) across various domains, pushing the boundaries of multi-modal applications. The emergence of Stable Diffusion XL in April 2023 brought image generation into the mainstream. The generation of videos from textual input poses a significant computational challenge within computer science. Nonetheless, recent advancements in text-to-video artificial intelligence (AI) technologies have showcased notable progress within this domain. [1] Following the rapid development of LLMs, OpenAI's release of the Sora promotional video in February 2024 marked a significant milestone in video generation. In this paper, we explore a novel approach to generating coherent videos by fine-tuning Stable Diffusion, enabling the production of temporally consistent video frames. Our architecture integrates LLM-based text expansion, multi-frame generation using Stable Diffusion, and final video synthesis. Experimental results demonstrate that our method significantly improves the temporal coherence of generated videos.
Read the paper
Related studies
- Learning from First-Generation Qualitative Approaches in the IS Discipline: An Evolutionary View and Some Implications for Authors and Evaluators (PART 1/2)
- Commentary on the clinical and preclinical dosage limits of interstitially administered magnetic fluids for therapeutic hyperthermia based on current practice and efficacy models
- Prosthesis: New Technological Opportunities and Innovative Biomedical Devices
- The history and future of neural modeling for cochlear implants
- Ryanodine receptors: waking up from refractoriness
- Novel Transgenic Rabbit Model Sheds Light on the Puzzling Role of Matrix Metalloproteinase-12 in Atherosclerosis
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.