Are synthetic clinical notes useful for real natural language processing tasks: A case study on clinical entity recognition

Jianfu Li, Yujia Zhou, Xiaoqian Jiang, Karthik Natarajan, Serguei Vs Pakhomov, Hongfang Liu, Hua Xu

Research output: Contribution to journalArticlepeer-review

23 Scopus citations

Abstract

Objective:: Developing clinical natural language processing systems often requires access to many clinical documents, which are not widely available to the public due to privacy and security concerns. To address this challenge, we propose to develop methods to generate synthetic clinical notes and evaluate their utility in real clinical natural language processing tasks. Materials and Methods:: We implemented 4 state-of-the-art text generation models, namely CharRNN, SegGAN, GPT-2, and CTRL, to generate clinical text for the History and Present Illness section. We then manually annotated clinical entities for randomly selected 500 History and Present Illness notes generated from the best-performing algorithm. To compare the utility of natural and synthetic corpora, we trained named entity recognition (NER) models from all 3 corpora and evaluated their performance on 2 independent natural corpora. Results:: Our evaluation shows GPT-2 achieved the best BLEU (bilingual evaluation understudy) score (with a BLEU-2 of 0.92). NER models trained on synthetic corpus generated by GPT-2 showed slightly better performance on 2 independent corpora: strict F1 scores of 0.709 and 0.748, respectively, when compared with the NER models trained on natural corpus (F1 scores of 0.706 and 0.737, respectively), indicating the good utility of synthetic corpora in clinical NER model development. In addition, we also demonstrated that an augmented method that combines both natural and synthetic corpora achieved better performance than that uses the natural corpus only. Conclusions:: Recent advances in text generation have made it possible to generate synthetic clinical notes that could be useful for training NER models for information extraction from natural clinical notes, thus lowering the privacy concern and increasing data availability. Further investigation is needed to apply this technology to practice.

Original languageEnglish (US)
Pages (from-to)2193-2201
Number of pages9
JournalJournal of the American Medical Informatics Association
Volume28
Issue number10
DOIs
StatePublished - Oct 1 2021

Bibliographical note

Funding Information:
This research was partially supported by the National Institutes of Health under Award Numbers U01TR002062, R01AI130460, R01GM103859, and U24CA194215.

Publisher Copyright:
© 2021 The Author(s).

Keywords

  • clinical notes
  • named entity recognition
  • natural language processing
  • neural language model
  • text generation

Fingerprint

Dive into the research topics of 'Are synthetic clinical notes useful for real natural language processing tasks: A case study on clinical entity recognition'. Together they form a unique fingerprint.

Cite this