TY - JOUR
T1 - Identifying Information Gaps in Electronic Health Records by Using Natural Language Processing
T2 - Gynecologic Surgery History Identification
AU - Moon, Sungrim
AU - Carlson, Luke A.
AU - Moser, Ethan D.
AU - Agnikula Kshatriya, Bhavani Singh
AU - Smith, Carin Y.
AU - Rocca, Walter A.
AU - Rocca, Liliana Gazzuola
AU - Bielinski, Suzette J.
AU - Liu, Hongfang
AU - Larson, Nicholas B.
N1 - Publisher Copyright:
©Sungrim Moon, Luke A Carlson, Ethan D Moser, Bhavani Singh Agnikula Kshatriya, Carin Y Smith, Walter A Rocca, Liliana Gazzuola Rocca, Suzette J Bielinski, Hongfang Liu, Nicholas B Larson.
PY - 2022/1
Y1 - 2022/1
N2 - Background: Electronic health records (EHRs) are a rich source of longitudinal patient data. However, missing information due to clinical care that predated the implementation of EHR system(s) or care that occurred at different medical institutions impedes complete ascertainment of a patient’s medical history. Objective: This study aimed to investigate information discrepancies and to quantify information gaps by comparing the gynecological surgical history extracted from an EHR of a single institution by using natural language processing (NLP) techniques with the manually curated surgical history information through chart review of records from multiple independent regional health care institutions. Methods: To facilitate high-throughput evaluation, we developed a rule-based NLP algorithm to detect gynecological surgery history from the unstructured narrative of the Mayo Clinic EHR. These results were compared to a gold standard cohort of 3870 women with gynecological surgery status adjudicated using the Rochester Epidemiology Project medical records–linkage system. We quantified and characterized the information gaps observed that led to misclassification of the surgical status. Results: The NLP algorithm achieved precision of 0.85, recall of 0.82, and F1-score of 0.83 in the test set (n=265) relative to outcomes abstracted from the Mayo EHR. This performance attenuated when directly compared to the gold standard (precision 0.79, recall 0.76, and F1-score 0.76), with the majority of misclassifications being false negatives in nature. We then applied the algorithm to the remaining patients (n=3340) and identified 2 types of information gaps through error analysis. First, 6% (199/3340) of women in this study had no recorded surgery information or partial information in the EHR. Second, 4.3% (144/3340) of women had inconsistent or inaccurate information within the clinical narrative owing to misinterpreted information, erroneous “copy and paste,” or incorrect information provided by patients. Additionally, the NLP algorithm misclassified the surgery status of 3.6% (121/3340) of women. Conclusions: Although NLP techniques were able to adequately recreate the gynecologic surgical status from the clinical narrative, missing or inaccurately reported and recorded information resulted in much of the misclassification observed. Therefore, alternative approaches to collect or curate surgical history are needed.
AB - Background: Electronic health records (EHRs) are a rich source of longitudinal patient data. However, missing information due to clinical care that predated the implementation of EHR system(s) or care that occurred at different medical institutions impedes complete ascertainment of a patient’s medical history. Objective: This study aimed to investigate information discrepancies and to quantify information gaps by comparing the gynecological surgical history extracted from an EHR of a single institution by using natural language processing (NLP) techniques with the manually curated surgical history information through chart review of records from multiple independent regional health care institutions. Methods: To facilitate high-throughput evaluation, we developed a rule-based NLP algorithm to detect gynecological surgery history from the unstructured narrative of the Mayo Clinic EHR. These results were compared to a gold standard cohort of 3870 women with gynecological surgery status adjudicated using the Rochester Epidemiology Project medical records–linkage system. We quantified and characterized the information gaps observed that led to misclassification of the surgical status. Results: The NLP algorithm achieved precision of 0.85, recall of 0.82, and F1-score of 0.83 in the test set (n=265) relative to outcomes abstracted from the Mayo EHR. This performance attenuated when directly compared to the gold standard (precision 0.79, recall 0.76, and F1-score 0.76), with the majority of misclassifications being false negatives in nature. We then applied the algorithm to the remaining patients (n=3340) and identified 2 types of information gaps through error analysis. First, 6% (199/3340) of women in this study had no recorded surgery information or partial information in the EHR. Second, 4.3% (144/3340) of women had inconsistent or inaccurate information within the clinical narrative owing to misinterpreted information, erroneous “copy and paste,” or incorrect information provided by patients. Additionally, the NLP algorithm misclassified the surgery status of 3.6% (121/3340) of women. Conclusions: Although NLP techniques were able to adequately recreate the gynecologic surgical status from the clinical narrative, missing or inaccurately reported and recorded information resulted in much of the misclassification observed. Therefore, alternative approaches to collect or curate surgical history are needed.
KW - Digital health
KW - EHealth
KW - Electronic health records
KW - Gynecologic surgery
KW - Gynecology
KW - Health information interoperability
KW - Information gap
KW - Medical informatics
KW - Natural language processing
KW - Surgery
UR - http://www.scopus.com/inward/record.url?scp=85123901123&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85123901123&partnerID=8YFLogxK
U2 - 10.2196/29015
DO - 10.2196/29015
M3 - Article
C2 - 35089141
AN - SCOPUS:85123901123
SN - 1439-4456
VL - 24
JO - Journal of medical Internet research
JF - Journal of medical Internet research
IS - 1
M1 - e29015
ER -