Psychometric Evaluation of the Cybersecurity Curriculum Assessment

Geoffrey L. Herman; Shan Huang; Peter A. Peterson; Linda Oliva; Enis Golaszewski; Alan T. Sherman

doi:10.1145/3545945.3569762

Psychometric Evaluation of the Cybersecurity Curriculum Assessment

Geoffrey L. Herman, Shan Huang, Peter A. Peterson, Linda Oliva, Enis Golaszewski, Alan T. Sherman

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Scopus citations

Abstract

We present a psychometric evaluation of the Cybersecurity Curriculum Assessment (CCA), completed by 193 students from seven colleges and universities. The CCA builds on our prior work developing and validating a Cybersecurity Concept Inventory (CCI), which measures students' conceptual understanding of cybersecurity after a first course in the area. The CCA deepens the conceptual complexity and technical depth expectations, assessing conceptual knowledge of students who had completed multiple courses in cybersecurity. We review our development of the CCA and present our evaluation of the instrument using Classical Test Theory and Item-Response Theory. The CCA is a difficult assessment, providing reliable measurements of student knowledge and deeper information about high-performing students.

Original language	English (US)
Title of host publication	SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education
Publisher	Association for Computing Machinery, Inc
Pages	228-234
Number of pages	7
ISBN (Electronic)	9781450394314
DOIs	https://doi.org/10.1145/3545945.3569762
State	Published - Mar 2 2023
Externally published	Yes
Event	54th ACM Technical Symposium on Computer Science Education, SIGCSE 2023 - Toronto, Canada Duration: Mar 15 2023 → Mar 18 2023

Publication series

Name	SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education
Volume	1

Conference

Conference	54th ACM Technical Symposium on Computer Science Education, SIGCSE 2023
Country/Territory	Canada
City	Toronto
Period	3/15/23 → 3/18/23

Bibliographical note

Funding Information:
We thank the many people who contributed to the CATS project as Delphi experts, interview subjects, Hackathon participants, expert reviewers, student subjects, and former team members, including Michael Neary, Spencer Offenberger, Geet Parekh, Konstantinos Patsourakos, Dhananjay Phatak, and Julia Thompson. Support for this research was provided in part by the U.S. Department of Defense under CAE-R grants H98230-15-1-0294, H98230-15-1-0273, H98230-17-1-0349, H98230-17-1-0347; and by the National Science Foundation under UMBC SFS grants DGE-1241576, 1753681, and SFS Capacity Grants DGE-1819521, 1820531.

Publisher Copyright:
© 2023 ACM.

Keywords

classical test theory
concept inventories
cybersecurity assessment tools (cats)
cybersecurity curriculum assessment
cybersecurity education
item response theory
psychometrics

Publisher link

10.1145/3545945.3569762

Find It

Find It at U of M Libraries

Cite this

Herman, G. L., Huang, S., Peterson, P. A., Oliva, L., Golaszewski, E., & Sherman, A. T. (2023). Psychometric Evaluation of the Cybersecurity Curriculum Assessment. In SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education (pp. 228-234). (SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education; Vol. 1). Association for Computing Machinery, Inc. https://doi.org/10.1145/3545945.3569762

Psychometric Evaluation of the Cybersecurity Curriculum Assessment. / Herman, Geoffrey L.; Huang, Shan; Peterson, Peter A. et al.
SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education. Association for Computing Machinery, Inc, 2023. p. 228-234 (SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education; Vol. 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Herman, GL, Huang, S, Peterson, PA, Oliva, L, Golaszewski, E & Sherman, AT 2023, Psychometric Evaluation of the Cybersecurity Curriculum Assessment. in SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education. SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education, vol. 1, Association for Computing Machinery, Inc, pp. 228-234, 54th ACM Technical Symposium on Computer Science Education, SIGCSE 2023, Toronto, Canada, 3/15/23. https://doi.org/10.1145/3545945.3569762

Herman GL, Huang S, Peterson PA, Oliva L, Golaszewski E, Sherman AT. Psychometric Evaluation of the Cybersecurity Curriculum Assessment. In SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education. Association for Computing Machinery, Inc. 2023. p. 228-234. (SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education). doi: 10.1145/3545945.3569762

Herman, Geoffrey L. ; Huang, Shan ; Peterson, Peter A. et al. / Psychometric Evaluation of the Cybersecurity Curriculum Assessment. SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education. Association for Computing Machinery, Inc, 2023. pp. 228-234 (SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education).

@inproceedings{b6c1cea591804df59210f54713b3fe65,

title = "Psychometric Evaluation of the Cybersecurity Curriculum Assessment",

abstract = "We present a psychometric evaluation of the Cybersecurity Curriculum Assessment (CCA), completed by 193 students from seven colleges and universities. The CCA builds on our prior work developing and validating a Cybersecurity Concept Inventory (CCI), which measures students' conceptual understanding of cybersecurity after a first course in the area. The CCA deepens the conceptual complexity and technical depth expectations, assessing conceptual knowledge of students who had completed multiple courses in cybersecurity. We review our development of the CCA and present our evaluation of the instrument using Classical Test Theory and Item-Response Theory. The CCA is a difficult assessment, providing reliable measurements of student knowledge and deeper information about high-performing students.",

keywords = "classical test theory, concept inventories, cybersecurity assessment tools (cats), cybersecurity curriculum assessment, cybersecurity education, item response theory, psychometrics",

author = "Herman, {Geoffrey L.} and Shan Huang and Peterson, {Peter A.} and Linda Oliva and Enis Golaszewski and Sherman, {Alan T.}",

note = "Funding Information: We thank the many people who contributed to the CATS project as Delphi experts, interview subjects, Hackathon participants, expert reviewers, student subjects, and former team members, including Michael Neary, Spencer Offenberger, Geet Parekh, Konstantinos Patsourakos, Dhananjay Phatak, and Julia Thompson. Support for this research was provided in part by the U.S. Department of Defense under CAE-R grants H98230-15-1-0294, H98230-15-1-0273, H98230-17-1-0349, H98230-17-1-0347; and by the National Science Foundation under UMBC SFS grants DGE-1241576, 1753681, and SFS Capacity Grants DGE-1819521, 1820531. Publisher Copyright: {\textcopyright} 2023 ACM.; 54th ACM Technical Symposium on Computer Science Education, SIGCSE 2023 ; Conference date: 15-03-2023 Through 18-03-2023",

year = "2023",

month = mar,

day = "2",

doi = "10.1145/3545945.3569762",

language = "English (US)",

series = "SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education",

publisher = "Association for Computing Machinery, Inc",

pages = "228--234",

booktitle = "SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education",

}

TY - GEN

T1 - Psychometric Evaluation of the Cybersecurity Curriculum Assessment

AU - Herman, Geoffrey L.

AU - Huang, Shan

AU - Peterson, Peter A.

AU - Oliva, Linda

AU - Golaszewski, Enis

AU - Sherman, Alan T.

N1 - Funding Information: We thank the many people who contributed to the CATS project as Delphi experts, interview subjects, Hackathon participants, expert reviewers, student subjects, and former team members, including Michael Neary, Spencer Offenberger, Geet Parekh, Konstantinos Patsourakos, Dhananjay Phatak, and Julia Thompson. Support for this research was provided in part by the U.S. Department of Defense under CAE-R grants H98230-15-1-0294, H98230-15-1-0273, H98230-17-1-0349, H98230-17-1-0347; and by the National Science Foundation under UMBC SFS grants DGE-1241576, 1753681, and SFS Capacity Grants DGE-1819521, 1820531. Publisher Copyright: © 2023 ACM.

PY - 2023/3/2

Y1 - 2023/3/2

N2 - We present a psychometric evaluation of the Cybersecurity Curriculum Assessment (CCA), completed by 193 students from seven colleges and universities. The CCA builds on our prior work developing and validating a Cybersecurity Concept Inventory (CCI), which measures students' conceptual understanding of cybersecurity after a first course in the area. The CCA deepens the conceptual complexity and technical depth expectations, assessing conceptual knowledge of students who had completed multiple courses in cybersecurity. We review our development of the CCA and present our evaluation of the instrument using Classical Test Theory and Item-Response Theory. The CCA is a difficult assessment, providing reliable measurements of student knowledge and deeper information about high-performing students.

AB - We present a psychometric evaluation of the Cybersecurity Curriculum Assessment (CCA), completed by 193 students from seven colleges and universities. The CCA builds on our prior work developing and validating a Cybersecurity Concept Inventory (CCI), which measures students' conceptual understanding of cybersecurity after a first course in the area. The CCA deepens the conceptual complexity and technical depth expectations, assessing conceptual knowledge of students who had completed multiple courses in cybersecurity. We review our development of the CCA and present our evaluation of the instrument using Classical Test Theory and Item-Response Theory. The CCA is a difficult assessment, providing reliable measurements of student knowledge and deeper information about high-performing students.

KW - classical test theory

KW - concept inventories

KW - cybersecurity assessment tools (cats)

KW - cybersecurity curriculum assessment

KW - cybersecurity education

KW - item response theory

KW - psychometrics

UR - http://www.scopus.com/inward/record.url?scp=85149828266&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85149828266&partnerID=8YFLogxK

U2 - 10.1145/3545945.3569762

DO - 10.1145/3545945.3569762

M3 - Conference contribution

AN - SCOPUS:85149828266

T3 - SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education

SP - 228

EP - 234

BT - SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education

PB - Association for Computing Machinery, Inc

T2 - 54th ACM Technical Symposium on Computer Science Education, SIGCSE 2023

Y2 - 15 March 2023 through 18 March 2023

ER -

Psychometric Evaluation of the Cybersecurity Curriculum Assessment

Abstract

Publication series

Conference

Bibliographical note

Keywords

Publisher link

Find It

Other files and links

Fingerprint

Cite this