Using discordance to improve classification in narrative clinical databases: An application to community-acquired pneumonia

George Hripcsak, Charles Knirsch, Li Zhou, Adam Wilcox, Genevieve B. Melton

Research output: Contribution to journalArticlepeer-review

18 Scopus citations

Abstract

Data mining in electronic medical records may facilitate clinical research, but much of the structured data may be miscoded, incomplete, or non-specific. The exploitation of narrative data using natural language processing may help, although nesting, varying granularity, and repetition remain challenges. In a study of community-acquired pneumonia using electronic records, these issues led to poor classification. Limiting queries to accurate, complete records led to vastly reduced, possibly biased samples. We exploited knowledge latent in the electronic records to improve classification. A similarity metric was used to cluster cases. We defined discordance as the degree to which cases within a cluster give different answers for some query that addresses a classification task of interest. Cases with higher discordance are more likely to be incorrectly classified, and can be reviewed manually to adjust the classification, improve the query, or estimate the likely accuracy of the query. In a study of pneumonia-in which the ICD9-CM coding was found to be very poor-the discordance measure was statistically significantly correlated with classification correctness (.45; 95% CI .15-.62).

Original languageEnglish (US)
Pages (from-to)296-304
Number of pages9
JournalComputers in Biology and Medicine
Volume37
Issue number3
DOIs
StatePublished - Mar 1 2007

Keywords

  • Classification
  • Community-acquired pneumonia
  • Data mining
  • Discordance
  • Electronic medical records
  • Natural language processing
  • Similarity metrics

Fingerprint Dive into the research topics of 'Using discordance to improve classification in narrative clinical databases: An application to community-acquired pneumonia'. Together they form a unique fingerprint.

Cite this