TCK-QA: Benchmarking Knowledge Graph-Augmented LLMs for Question Answering on the Turkish Penal Code
Abstract
Abstract In this study, we propose a knowledge graph-based question-answering system for the Turkish Penal Code (TCK, Law No. 5237). The system has two phases, namely an offline indexing pipeline and an online question-answering (QA) pipeline. In the indexing phase, we use GPT-4o to build a Neo4j knowledge graph from the 345 articles. The graph contains 1572 entities of seven types and 2366 relationships of nine types. In the QA phase, a six-stage GraphRAG pipeline answers each user query. The pipeline consists of query analysis, entity retrieval, graph traversal, context reranking, context construction, and answer generation. One major challenge in this study was the lack of a standard benchmark for Turkish legal RAG systems. Therefore, we introduce the Turkish Penal Code question-answering dataset (TCK-QA), a novel dataset for the evaluation of legal information retrieval systems on Turkish criminal law. The dataset contains 150 questions in single-hop, multi-hop, and reasoning types, and the reference answers are taken from the TCK article texts and reviewed by a legal domain expert. We hope that the dataset will encourage further research on question-answering for Turkish legal texts. We perform extensive analyses on the dataset using seven RAGAS metrics and three custom article-level retrieval metrics. GraphRAG outperforms BasicRAG on 114 of 150 questions (76%), with an article F1 of 0.61 against 0.25. The largest answer-quality gain appears in reasoning questions,…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.