Bilingual Retrieval-Augmented Access to ANVISA Regulatory Documents: A Technical Evaluation Using Drug-Development and Health-Product Questions
Abstract
Abstract Background. Regulatory teams often need to interpret long health-authority documents while working across languages. In Brazil, many regulatory sources are published in Portuguese, while companies and scientists may ask questions in English. Objective. To evaluate whether a bilingual retrieval-augmented generation prototype could locate, cite, and summarize selected ANVISA regulatory requirements relevant to drug development and health-product regulation. Methods. Six ANVISA regulatory documents were cleaned and indexed as an article-level closed corpus. The system was implemented in Python 3.10 using LlamaIndex, sentence-transformers, rank-bm25, Streamlit, and the Google Generative AI interface. Dense retrieval used sentence-transformers/all-MiniLM-L6-v2; keyword retrieval used BM25; candidates were merged by reciprocal rank fusion and reranked with cross-encoder/mmarco-mMiniLMv2-L12-H384-v1. The final answer model was models/gemini-flash-lite-latest. The benchmark included 200 bilingual questions: 70 Portuguese answerable, 70 English answerable, 20 multi-article or exception questions, and 40 unanswerable or adversarial questions. An automated AI judge scored faithfulness, citation correctness, completeness, and refusal correctness on a 0 to 1 scale. Results. The final article-aware index contained 382 detected regulatory articles and 481 chunks. In the first AI-authored prospective stress test, answerable questions had source hit 1.000, source MRR 0.988,…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.