MKRAG: Medical Knowledge Retrieval Augmented Generation for Medical Question Answering

Presentation Time: 09:30 AM - 09:45 AM

Abstract Keywords: Usability, Deep Learning, Large Language Models (LLMs), Knowledge Representation and Information Modeling
Primary Track: Applications

Large Language Models (LLMs), although powerful in general domains, often perform poorly on domain-specific tasks like medical question answering (QA). Moreover, they tend to function as "black-boxes," making it challenging to modify their behavior. To address the problem, our study delves into retrieval augmented generation (RAG), aiming to improve LLM responses without the need for fine-tuning or retraining. Specifically, we propose a comprehensive retrieval strategy to extract medical facts from an external knowledge base, and then inject them into the query prompt for LLMs. Focusing on medical QA using the MedQA-SMILE dataset, we evaluate the impact of different retrieval models and the number of facts provided to the LLM. Notably, our retrieval-augmented Vicuna-7B model exhibited an accuracy improvement from 44.46% to 48.54%. This work underscores the potential of RAG to enhance LLM performance, offering a practical approach to mitigate the challenges of black-box LLMs.

Speaker(s):
Yucheng Shi, Ph.D. Student
University of Georgia

Author(s):
Yucheng Shi, Ph.D. student - University of Georgia; Shaochen Xu, Ph.D. student - University of Georgia; Tianze Yang, Ph.D. student - University of Georgia; Zhengliang Liu; Tianming Liu, Ph.D. - University of Georgia; Xiang Li, PhD - Massachusetts General Hospital and Harvard Medical School; Ninghao Liu, Ph.D. - University of Georgia;