Diffusion-Based Adversarial Generation with SAM-Guided Spatial Semantics for Text-to-Image Models
Abstract
While adversarial examples have been extensively studied, most mainstream approaches focus on perturbations in the pixel space, which often introduce high-frequency artifacts and suffer from limited cross-model transferability. Moreover, these pixel-level modifications typically ignore the semantic structure of the image, reducing the effectiveness of targeted attacks. To address these limitations, we explore adversarial generation in the latent space of diffusion models, guided by semantic masks from the Segment Anything Model (SAM). In this work, we propose an innovative strategy that leverages large-scale text-to-image generative models. By integrating adversarial optimization with reversible diffusion inference (DDIM Inversion), we produce adversarial examples that exhibit both high fidelity and robust attacking performance. Meanwhile, we employ the Segment Anything Model (SAM) for semantic parsing to deliver fine-grained spatial guidance during the text-to-image generation process, enabling more effective perturbations in local regions.Extensive experiments demonstrate that our proposed method significantly enhances the adversarial potential of generated examples across diverse architectures and defense settings. For instance, on black-box models such as Swin-B and DeiT-B, our method achieves up to 45.8% and 51.4% attack success rates, respectively, outperforming other diffusion-based methods like DiffAttack by margins of 5% and 1.5%, while maintaining superior…
Read the paper
Related studies
- Learning from First-Generation Qualitative Approaches in the IS Discipline: An Evolutionary View and Some Implications for Authors and Evaluators (PART 1/2)
- Commentary on the clinical and preclinical dosage limits of interstitially administered magnetic fluids for therapeutic hyperthermia based on current practice and efficacy models
- Prosthesis: New Technological Opportunities and Innovative Biomedical Devices
- The history and future of neural modeling for cochlear implants
- Ryanodine receptors: waking up from refractoriness
- Novel Transgenic Rabbit Model Sheds Light on the Puzzling Role of Matrix Metalloproteinase-12 in Atherosclerosis
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.