Towards interpretable image generation assessment: a exploration for structured multimodal approach
Abstract
Recent advances in diffusion models have led to impressive progress in text-to-image generation. However, while the visual fidelity of these models continues to improve, their semantic alignment with the input prompts—particularly in terms of object presence, relationships, and stylistic adherence—remains insufficiently evaluated. Existing automatic metrics such as CLIP-Score and FID focus on global embeddings and low- level statistics, failing to capture nuanced semantic correctness. This work presents a semantic evaluation framework for diffusion-generated images, powered by Multimodal Large Language Models (MLLMs) such as LLaVA-1.6/LLaVA- ov and Qwen2.5-VL. The workflow begins by extracting semantic dimensions (objects, attributes, relationships, styles) from prompts using NLP parsing and Semantic Role Labelling. It then dynamically constructs evaluation questions, guiding MLLMs to perform fine-grained image assessments in structured JSON format. From these responses, quantitative metrics are computed to evaluate semantic coverage, relation validity, stylistic consistency, and logical soundness. Experiments were conducted on images generated by SD (Stable Diffusion) model (SD-1.5 and SD-XL), along with three fine-tuned LoRA variants for each. The results demonstrate the framework’s ability to detect meaningful differences aligned with human judgment: SD-XL outperforms SD-1.5 in semantic coverage and relation validity by an average margin of 0.05–0.08 across all MLLM…
Read the paper
Related studies
- Learning from First-Generation Qualitative Approaches in the IS Discipline: An Evolutionary View and Some Implications for Authors and Evaluators (PART 1/2)
- Commentary on the clinical and preclinical dosage limits of interstitially administered magnetic fluids for therapeutic hyperthermia based on current practice and efficacy models
- Prosthesis: New Technological Opportunities and Innovative Biomedical Devices
- The history and future of neural modeling for cochlear implants
- Ryanodine receptors: waking up from refractoriness
- Novel Transgenic Rabbit Model Sheds Light on the Puzzling Role of Matrix Metalloproteinase-12 in Atherosclerosis
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.