ChatGPT Responses to Questions About Clinical Decision Making: A Preliminary Analysis of Accuracy and Completeness

PURPOSE: The primary aim of this preliminary study was to examine the potential advantages and disadvantages of using ChatGPT-3.5 (free version) to obtain information pertaining to the discipline of communication sciences and disorders (CSD).

METHOD: Nine university-based CSD researchers with expertise in different topic areas created questions of varying complexity (basic, advanced, evidence-based practice [EBP], and general CSD) to ask ChatGPT and then rated the software's responses for accuracy (6-point Likert scale) and completeness (3-point Likert scale).

RESULTS: Across all questions ( n = 162), the median accuracy was 5 (nearly all correct), with an M of 5.01. Regarding completeness, the median was 3 (comprehensive), and the mean was 2.41. Furthermore, there were no significant differences in composite ratings (accuracy and completeness combined) between responses to basic, advanced, and EBP questions; however, responses to general CSD questions received significantly higher composite ratings than responses to basic, advanced, and EBP questions. Despite strong performance of ChatGPT on average, the distribution of accuracy and completeness ratings showed that more than half of ChatGPT's responses were rated as not fully accurate, only 45.68% of the responses received a rating of 6 (correct). On completeness ratings, only 54.32% of responses received a rating of 3 (comprehensive).

CONCLUSIONS: Our results indicate that ChatGPT can quickly provide answers and often generate responses that experts judge as accurate. However, clinicians cannot depend on ChatGPT for accurate and complete responses, especially for complex clinical questions. Using ChatGPT for clinical decision making may be particularly risky for inexperienced or student clinicians, who may have difficulty spotting inaccuracies.

SUPPLEMENTAL MATERIAL: https://doi.org/10.23641/asha.29565167.