AI for arbitrary skeleton guided text-to-image generation
Abstract
Autoregressive (AR) image generation models, for example LlamaGen, achieve generative quality as good as diffusion models in text-to-image tasks through single-stage token prediction. However, many applications need spatial conditions, such as information about human pose or skeleton, in addition to text captions. Current pose-guided text-to-image approaches are mainly based on diffusion models (e.g. HumanSD, Stable-Pose), while systematic control over pose keypoints for AR frameworks and fair comparison under the same data and metrics are still relatively limited. This dissertation focuses on this rarely explored research area by proposing Pose-LlamaGen, a skeleton-guided text-to-image method based on an open-source AR model called LlamaGen. Pose-LlamaGen only injects pose-conditioned residuals at image token positions in a series of carefully selected backbone layers, keeping the weights in LlamaGen backbone frozen and preserving the decoding procedure in a single stage. The encoder for pose keypoint images maps every skeleton image, which is annotated by OpenPose in this dissertation, to a token sequence aligned with the image token grid, while a ControlNet-style bypass network with zero-initialized projections produces residuals for target layers in LlamaGen. Only pose encoder and bypass network are trained, ensuring the pose control mechanism in Pose-LlamaGen parameter-efficient and computationally light. The dissertation evaluates multiple backbones as the pose encoder…
Read the paper
Related studies
- Learning from First-Generation Qualitative Approaches in the IS Discipline: An Evolutionary View and Some Implications for Authors and Evaluators (PART 1/2)
- Commentary on the clinical and preclinical dosage limits of interstitially administered magnetic fluids for therapeutic hyperthermia based on current practice and efficacy models
- Prosthesis: New Technological Opportunities and Innovative Biomedical Devices
- The history and future of neural modeling for cochlear implants
- Ryanodine receptors: waking up from refractoriness
- Novel Transgenic Rabbit Model Sheds Light on the Puzzling Role of Matrix Metalloproteinase-12 in Atherosclerosis
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.