Measuring the Security of LLM Applications: A Taxonomy and Empirical Evaluation of Attacks and Defense Tools
Abstract
Large Language Models (LLMs) are rapidly becoming foundational components of modern software systems, enabling applications such as conversational agents, software copilots, knowledge retrieval platforms, and autonomous AI assistants. However, their integration into real-world environments introduces a new class of security and privacy risks that extend beyond traditional machine learning vulnerabilities. These risks arise from LLM-specific characteristics including instruction-following behavior, retrieval-augmented generation, tool invocation, and interaction with external data sources. This paper presents a structured analysis of the evolving security threat landscape targeting LLM-based systems. We develop a taxonomy of security threats that categorizes attacks into five major classes: input manipulation attacks, training and model manipulation attacks, confidentiality and privacy violations, availability attacks, and supply-chain vulnerabilities. The taxonomy highlights how adversaries exploit different layers of the LLM application pipeline, from user prompts and retrieval systems to training data and deployment infrastructure. In addition, we examine the emerging ecosystem of LLM security tools designed to detect and mitigate these threats, including red-teaming frameworks, prompt and output filtering systems, observability platforms, and enterprise security solutions. By mapping these defenses to the identified threat categories, we identify important gaps in current…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.