Percentage Agreement Between Individual LLMs and Certified Human Coder in Extraction of ICD-10-CM Codes

Poster Number: P44
Presentation Time: 05:00 PM - 06:30 PM

Abstract Keywords: Large Language Models (LLMs), Deep Learning, Human-computer Interaction, Information Extraction
Primary Track: Foundations
Programmatic Theme: Clinical Informatics

The International Classification of Diseases- tenth revision – clinical modification (ICD-10-CM) is a complex code system. This study assessed the capability of ChatGPT 3.5, ChatGPT 4, Claude 2.1 and Claude 3 to extract these codes from unstructured clinical notes in comparison to a certified human coder. Based on our findings, we observed minimal to no agreement between LLMs and the certified human coder.

Speaker(s):
Megan McDougal, MS
West Virginia University

Ashley Simmons, MBA
West Virginia University

Author(s):
Ankit Sakhuja, MBBS, MS - Icahn School of Medicine at Mount Sinai; Kullaya Takkavatakarn, MD - Icahn School of Medicine at Mount Sinai;