Small but Specialized: A Domain-Adapted Retrieval-Augmented LLM Outperforms Frontier Generalists in Pediatric and Adolescent Gynecology
Abstract
ABSTRACT:
Background: Pediatric and adolescent gynecology (PAG) is a highly specialized field where timely, accurate clinical guidance can prevent serious harm to a vulnerable population, yet most general-purpose large language models (LLMs) lack reliable, citation-grounded knowledge of this domain.
Objective: To develop and evaluate a domain-specialized AI model that provides safe, evidence-anchored answers to PAG clinical questions, and to test whether a small, specialized model can outperform large general-purpose models.
Methods: We fine-tuned the Mistral-7B-Instruct LLM using QLoRA on a curated question–answer corpus derived from a medical textbook, including selected chapters focused on pediatric and adolescent gynecology and gynecologic care for girls under 18 years old, and combined it with a retrieval-augmented generation (RAG) layer using BGE embeddings and FAISS over 250-word textbook chunks with chapter-level citation. Performance was assessed on 182 held-out questions from chapters never seen during training (no data leakage), using a comprehensive ten-metric evaluation suite spanning lexical overlap (ROUGE-1/2/L, BLEU), character-level fidelity (chrF++), token-level paraphrase robustness (METEOR), contextual embedding similarity (BERTScore), learned semantic similarity (BLEURT, SAS), and an LLM-as-judge clinical rubric (G-Eval), plus a faithfulness proxy.
Results: PAG-Health-LLM achieved BERTScore 0.909, ROUGE-L 0.413, METEOR 0.526, chrF++ 0.489, and BLEURT…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.