Beyond Generating Code: Evaluating GPT on a Data Visualization Course

Zhutian Chen, Chenyang Zhang, Qianwen Wang, Jakob Troidl, Simon Warchol, Johanna Beyer, Nils Gehlenborg, Hanspeter Pfister

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper presents an empirical evaluation of the performance of the Generative Pre-trained Transformer (GPT) model in Harvard's CS171 data visualization course. While previous studies have focused on GPT's ability to generate code for visualizations, this study goes beyond code generation to evaluate GPT's abilities in various visualization tasks, such as data interpretation, visualization design, visual data exploration, and insight communication. The evaluation utilized GPT-3.5 and GPT-4 through the APIs of OpenAI to complete assignments of CS171, and included a quantitative assessment based on the established course rubrics, a qualitative analysis informed by the feedback of three experienced graders, and an exploratory study of GPT's capabilities in completing border visualization tasks. Findings show that GPT-4 scored 80% on quizzes and homework, and Teaching Fellows could distinguish between GPT-and human-generated homework with 70% accuracy. The study also demonstrates GPT's potential in completing various visualization tasks, such as data cleanup, interaction with visualizations, and insight communication. The paper concludes by discussing the strengths and limitations of GPT in data visualization, potential avenues for incorporating GPT in broader visualization tasks, and the need to redesign visualization education.

Original languageEnglish (US)
Title of host publicationProceedings - 2023 IEEE VIS Workshop on Visualization Education, Literacy, and Activities, EduVis 2023
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages16-21
Number of pages6
ISBN (Electronic)9798350330304
DOIs
StatePublished - 2023
Externally publishedYes
Event1st IEEE VIS Workshop on Visualization Education, Literacy, and Activities, EduVis 2023 - Hybrid, Melbourne, Australia
Duration: Oct 22 2023Oct 23 2023

Publication series

NameProceedings - 2023 IEEE VIS Workshop on Visualization Education, Literacy, and Activities, EduVis 2023

Conference

Conference1st IEEE VIS Workshop on Visualization Education, Literacy, and Activities, EduVis 2023
Country/TerritoryAustralia
CityHybrid, Melbourne
Period10/22/2310/23/23

Bibliographical note

Publisher Copyright:
© 2023 IEEE.

Fingerprint

Dive into the research topics of 'Beyond Generating Code: Evaluating GPT on a Data Visualization Course'. Together they form a unique fingerprint.

Cite this