Safety Evaluation of Language Models for Norwegian Deployment - Simula Safety Report 1
Abstract
This report presents a comparative safety evaluation of five language models on 36 Norwegian-specific scenarios, focusing on the disconnect between standard capability benchmarks and actual safety outcomes. We evaluated GPT-5, Borealis 4B, Gemma 3 4B, Llama 3.1 8B, and Qwen3 8B using the SimpleAudit framework across healthcare, legal, and public sector domains. The report identifies an "Understanding Paradox" among smaller models: performance on the EuroEval NLU benchmark was inversely related to safety. Qwen3 8B, the highest-ranked small model (#116), exhibited the worst safety profile with 16 critical failures, while Borealis 4B, the lowest-ranked (#249), achieved the best baseline safety. Furthermore, the evaluation demonstrates that Retrieval-Augmented Generation (RAG) is not a universal safeguard; while it aided high-failure models, it worsened specific healthcare scenarios for Llama 3.1 8B and Borealis 4B BF16. The report also details how higher precision (BF16) does not consistently reduce critical failures compared to quantized variants.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.