Data-Driven Investigation into Variants of Code Writing Questions

Liia Butler; Geoffrey Challen; Tao Xie

doi:10.1109/CSEET49119.2020.9206195

Data-Driven Investigation into Variants of Code Writing Questions

Liia Butler
, Geoffrey Challen
, Tao Xie

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

6 Scopus citations

Abstract

To defend against collaborative cheating in code writing questions, instructors of courses with online, asynchronous exams can use the strategy of question variants. These question variants are manually written questions to be selected at random during exam time to assess the same learning goal. In order to create these variants, currently the instructors have to rely on intuition to accomplish the competing goals of ensuring that variants are different enough to defend against collaborative cheating, and yet similar enough where students are assessed fairly. In this paper, we propose data-driven investigation into these variants. We apply our data-driven investigation into a dataset of three midterm exams from a large introductory programming course. Our results show that (1) observable inequalities of student performance exist between variants and (2) these differences are not just limited to score. Our results also show that the information gathered from our data-driven investigation can be used to provide recommendations for improving design of future variants.

Original language	English (US)
Title of host publication	2020 IEEE 32nd Conference on Software Engineering Education and Training, CSEE and T 2020
Editors	Marian Daun, Elke Hochmuller, Stephan Krusche, Bernd Brugge, Bastian Tenbergen
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	75-84
Number of pages	10
ISBN (Electronic)	9781728168074
DOIs	https://doi.org/10.1109/CSEET49119.2020.9206195
State	Published - Nov 2020
Externally published	Yes
Event	32nd IEEE Conference on Software Engineering Education and Training, CSEE and T 2020 - Munich, Germany Duration: Nov 9 2020 → Nov 12 2020

Publication series

Name	2020 IEEE 32nd Conference on Software Engineering Education and Training, CSEE and T 2020

Conference

Conference	32nd IEEE Conference on Software Engineering Education and Training, CSEE and T 2020
Country/Territory	Germany
City	Munich
Period	11/9/20 → 11/12/20

Bibliographical note

Publisher Copyright:
© 2020 IEEE.

Keywords

assessment
code writing
exam questions
Question variants

Publisher link

10.1109/CSEET49119.2020.9206195

Find It

Find It at U of M Libraries

Cite this

Butler, L., Challen, G., & Xie, T. (2020). Data-Driven Investigation into Variants of Code Writing Questions. In M. Daun, E. Hochmuller, S. Krusche, B. Brugge, & B. Tenbergen (Eds.), 2020 IEEE 32nd Conference on Software Engineering Education and Training, CSEE and T 2020 (pp. 75-84). Article 9206195 (2020 IEEE 32nd Conference on Software Engineering Education and Training, CSEE and T 2020). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CSEET49119.2020.9206195

Data-Driven Investigation into Variants of Code Writing Questions. / Butler, Liia; Challen, Geoffrey; Xie, Tao.
2020 IEEE 32nd Conference on Software Engineering Education and Training, CSEE and T 2020. ed. / Marian Daun; Elke Hochmuller; Stephan Krusche; Bernd Brugge; Bastian Tenbergen. Institute of Electrical and Electronics Engineers Inc., 2020. p. 75-84 9206195 (2020 IEEE 32nd Conference on Software Engineering Education and Training, CSEE and T 2020).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Butler, L, Challen, G & Xie, T 2020, Data-Driven Investigation into Variants of Code Writing Questions. in M Daun, E Hochmuller, S Krusche, B Brugge & B Tenbergen (eds), 2020 IEEE 32nd Conference on Software Engineering Education and Training, CSEE and T 2020., 9206195, 2020 IEEE 32nd Conference on Software Engineering Education and Training, CSEE and T 2020, Institute of Electrical and Electronics Engineers Inc., pp. 75-84, 32nd IEEE Conference on Software Engineering Education and Training, CSEE and T 2020, Munich, Germany, 11/9/20. https://doi.org/10.1109/CSEET49119.2020.9206195

Butler L, Challen G, Xie T. Data-Driven Investigation into Variants of Code Writing Questions. In Daun M, Hochmuller E, Krusche S, Brugge B, Tenbergen B, editors, 2020 IEEE 32nd Conference on Software Engineering Education and Training, CSEE and T 2020. Institute of Electrical and Electronics Engineers Inc. 2020. p. 75-84. 9206195. (2020 IEEE 32nd Conference on Software Engineering Education and Training, CSEE and T 2020). doi: 10.1109/CSEET49119.2020.9206195

Butler, Liia ; Challen, Geoffrey ; Xie, Tao. / Data-Driven Investigation into Variants of Code Writing Questions. 2020 IEEE 32nd Conference on Software Engineering Education and Training, CSEE and T 2020. editor / Marian Daun ; Elke Hochmuller ; Stephan Krusche ; Bernd Brugge ; Bastian Tenbergen. Institute of Electrical and Electronics Engineers Inc., 2020. pp. 75-84 (2020 IEEE 32nd Conference on Software Engineering Education and Training, CSEE and T 2020).

@inproceedings{d62b31f02d5d46169510a1cc2fb52bf3,

title = "Data-Driven Investigation into Variants of Code Writing Questions",

abstract = "To defend against collaborative cheating in code writing questions, instructors of courses with online, asynchronous exams can use the strategy of question variants. These question variants are manually written questions to be selected at random during exam time to assess the same learning goal. In order to create these variants, currently the instructors have to rely on intuition to accomplish the competing goals of ensuring that variants are different enough to defend against collaborative cheating, and yet similar enough where students are assessed fairly. In this paper, we propose data-driven investigation into these variants. We apply our data-driven investigation into a dataset of three midterm exams from a large introductory programming course. Our results show that (1) observable inequalities of student performance exist between variants and (2) these differences are not just limited to score. Our results also show that the information gathered from our data-driven investigation can be used to provide recommendations for improving design of future variants.",

keywords = "assessment, code writing, exam questions, Question variants",

author = "Liia Butler and Geoffrey Challen and Tao Xie",

note = "Publisher Copyright: {\textcopyright} 2020 IEEE.; 32nd IEEE Conference on Software Engineering Education and Training, CSEE and T 2020 ; Conference date: 09-11-2020 Through 12-11-2020",

year = "2020",

month = nov,

doi = "10.1109/CSEET49119.2020.9206195",

language = "English (US)",

series = "2020 IEEE 32nd Conference on Software Engineering Education and Training, CSEE and T 2020",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "75--84",

editor = "Marian Daun and Elke Hochmuller and Stephan Krusche and Bernd Brugge and Bastian Tenbergen",

booktitle = "2020 IEEE 32nd Conference on Software Engineering Education and Training, CSEE and T 2020",

}

TY - GEN

T1 - Data-Driven Investigation into Variants of Code Writing Questions

AU - Butler, Liia

AU - Challen, Geoffrey

AU - Xie, Tao

PY - 2020/11

Y1 - 2020/11

N2 - To defend against collaborative cheating in code writing questions, instructors of courses with online, asynchronous exams can use the strategy of question variants. These question variants are manually written questions to be selected at random during exam time to assess the same learning goal. In order to create these variants, currently the instructors have to rely on intuition to accomplish the competing goals of ensuring that variants are different enough to defend against collaborative cheating, and yet similar enough where students are assessed fairly. In this paper, we propose data-driven investigation into these variants. We apply our data-driven investigation into a dataset of three midterm exams from a large introductory programming course. Our results show that (1) observable inequalities of student performance exist between variants and (2) these differences are not just limited to score. Our results also show that the information gathered from our data-driven investigation can be used to provide recommendations for improving design of future variants.

AB - To defend against collaborative cheating in code writing questions, instructors of courses with online, asynchronous exams can use the strategy of question variants. These question variants are manually written questions to be selected at random during exam time to assess the same learning goal. In order to create these variants, currently the instructors have to rely on intuition to accomplish the competing goals of ensuring that variants are different enough to defend against collaborative cheating, and yet similar enough where students are assessed fairly. In this paper, we propose data-driven investigation into these variants. We apply our data-driven investigation into a dataset of three midterm exams from a large introductory programming course. Our results show that (1) observable inequalities of student performance exist between variants and (2) these differences are not just limited to score. Our results also show that the information gathered from our data-driven investigation can be used to provide recommendations for improving design of future variants.

KW - assessment

KW - code writing

KW - exam questions

KW - Question variants

UR - https://www.scopus.com/pages/publications/85097648378

UR - https://www.scopus.com/pages/publications/85097648378#tab=citedBy

U2 - 10.1109/CSEET49119.2020.9206195

DO - 10.1109/CSEET49119.2020.9206195

M3 - Conference contribution

AN - SCOPUS:85097648378

T3 - 2020 IEEE 32nd Conference on Software Engineering Education and Training, CSEE and T 2020

SP - 75

EP - 84

BT - 2020 IEEE 32nd Conference on Software Engineering Education and Training, CSEE and T 2020

A2 - Daun, Marian

A2 - Hochmuller, Elke

A2 - Krusche, Stephan

A2 - Brugge, Bernd

A2 - Tenbergen, Bastian

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 32nd IEEE Conference on Software Engineering Education and Training, CSEE and T 2020

Y2 - 9 November 2020 through 12 November 2020

ER -

Data-Driven Investigation into Variants of Code Writing Questions

Abstract

Publication series

Conference

Bibliographical note

Keywords

Publisher link

Find It

Other files and links

Fingerprint

Cite this