ChatGPT in Undergraduate Education: Performance of GPT-3.5 and Identification of AI-Generated Text in Introductory Neuroscience

ChatGPT and other large language models (LLMs) have the potential to significantly disrupt common educational practices and assessments, given their capability to quickly generate human-like text in response to user prompts. LLMs GPT-3.5 and GPT-4 have been tested against many standardized and high-stakes assessment materials (e.g. SAT, Uniform Bar Exam, GRE), demonstrating impressive but variable performance. Fewer studies have examined the performance of ChatGPT on course-level educational materials in ecologically-valid grading contexts. Here, we examine the performance of GPT-3.5 on undergraduate course materials and assess the ability of teaching assistants to identify AI-generated responses interleaved with student work. GPT-3.5 was prompted to respond to questions drawn from undergraduate neuroscience assessments. These AI-generated responses were interleaved with student-authored responses and graded blindly using existing course rubrics. In addition, a subset of responses were rated for their humanlikeness by teaching assistants who were blind to author status (AI vs. student). In general, GPT-3.5 performed within one standard deviation of the class average, but there were cases in which ChatGPT-generated responses substantially outperformed or underperformed relative to student responses. Teaching assistants who were blind to author status were able to identify ChatGPT-generated responses with better than chance accuracy, and those with personal experience using ChatGPT were significantly more accurate than those without ChatGPT experience. Despite high levels of identification accuracy, none of the teaching assistant raters endorsed sufficient confidence in their identifications to support reporting the response as an instance of academic dishonesty in a real-world classroom setting.