DuluthNLP at SemEval-2022 Task 7: Classifying Plausible Alternatives with Pre-trained ELECTRA

Samuel Akrah; Ted Pedersen

DuluthNLP at SemEval-2022 Task 7: Classifying Plausible Alternatives with Pre-trained ELECTRA

Computer Science (Duluth)

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

This paper describes the DuluthNLP system that participated in Task 7 of SemEval-2022 on Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts. Given an instructional text with an omitted token, the task requires models to classify or rank the plausibility of potential fillers. To solve the task, we fine-tuned the models BERT, RoBERTa, and ELECTRA on training data where potential fillers are rated for plausibility. This is a challenging problem, as shown by BERT-based models achieving accuracy less than 45%. However, our ELECTRA model with tuned class weights on CrossEntropyLoss achieves an accuracy of 53.3% on the official evaluation test data, which ranks 6 out of the 8 total submissions for Subtask A.

Original language	English (US)
Title of host publication	SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop
Editors	Guy Emerson, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh, Shyam Ratan
Publisher	Association for Computational Linguistics (ACL)
Pages	1062-1066
Number of pages	5
ISBN (Electronic)	9781955917803
State	Published - 2022
Event	16th International Workshop on Semantic Evaluation, SemEval 2022 - Seattle, United States Duration: Jul 14 2022 → Jul 15 2022

Publication series

Name	SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop

Conference

Conference	16th International Workshop on Semantic Evaluation, SemEval 2022
Country/Territory	United States
City	Seattle
Period	7/14/22 → 7/15/22

Bibliographical note

Funding Information:
The authors would like to thank the organizers for the opportunity to participate in SemEval-2021 Task 7. We are also grateful to the three anonymous reviewers for their thoughtful comments and feedback. Finally, we would like to thank Dr. Alexis Elder for her contributions regarding the ethical considerations of this work.

Publisher Copyright:
© 2022 Association for Computational Linguistics.

Find It

Find It at U of M Libraries

Cite this

Akrah, S., & Pedersen, T. (2022). DuluthNLP at SemEval-2022 Task 7: Classifying Plausible Alternatives with Pre-trained ELECTRA. In G. Emerson, N. Schluter, G. Stanovsky, R. Kumar, A. Palmer, N. Schneider, S. Singh, & S. Ratan (Eds.), SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop (pp. 1062-1066). (SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop). Association for Computational Linguistics (ACL).

DuluthNLP at SemEval-2022 Task 7: Classifying Plausible Alternatives with Pre-trained ELECTRA. / Akrah, Samuel; Pedersen, Ted.
SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop. ed. / Guy Emerson; Natalie Schluter; Gabriel Stanovsky; Ritesh Kumar; Alexis Palmer; Nathan Schneider; Siddharth Singh; Shyam Ratan. Association for Computational Linguistics (ACL), 2022. p. 1062-1066 (SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Akrah, S & Pedersen, T 2022, DuluthNLP at SemEval-2022 Task 7: Classifying Plausible Alternatives with Pre-trained ELECTRA. in G Emerson, N Schluter, G Stanovsky, R Kumar, A Palmer, N Schneider, S Singh & S Ratan (eds), SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop. SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop, Association for Computational Linguistics (ACL), pp. 1062-1066, 16th International Workshop on Semantic Evaluation, SemEval 2022, Seattle, United States, 7/14/22.

Akrah S, Pedersen T. DuluthNLP at SemEval-2022 Task 7: Classifying Plausible Alternatives with Pre-trained ELECTRA. In Emerson G, Schluter N, Stanovsky G, Kumar R, Palmer A, Schneider N, Singh S, Ratan S, editors, SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop. Association for Computational Linguistics (ACL). 2022. p. 1062-1066. (SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop).

Akrah, Samuel ; Pedersen, Ted. / DuluthNLP at SemEval-2022 Task 7 : Classifying Plausible Alternatives with Pre-trained ELECTRA. SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop. editor / Guy Emerson ; Natalie Schluter ; Gabriel Stanovsky ; Ritesh Kumar ; Alexis Palmer ; Nathan Schneider ; Siddharth Singh ; Shyam Ratan. Association for Computational Linguistics (ACL), 2022. pp. 1062-1066 (SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop).

@inproceedings{1c0a3bc7b49a4b24a8f52b2cf7c1db6d,

title = "DuluthNLP at SemEval-2022 Task 7: Classifying Plausible Alternatives with Pre-trained ELECTRA",

abstract = "This paper describes the DuluthNLP system that participated in Task 7 of SemEval-2022 on Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts. Given an instructional text with an omitted token, the task requires models to classify or rank the plausibility of potential fillers. To solve the task, we fine-tuned the models BERT, RoBERTa, and ELECTRA on training data where potential fillers are rated for plausibility. This is a challenging problem, as shown by BERT-based models achieving accuracy less than 45%. However, our ELECTRA model with tuned class weights on CrossEntropyLoss achieves an accuracy of 53.3% on the official evaluation test data, which ranks 6 out of the 8 total submissions for Subtask A.",

author = "Samuel Akrah and Ted Pedersen",

note = "Funding Information: The authors would like to thank the organizers for the opportunity to participate in SemEval-2021 Task 7. We are also grateful to the three anonymous reviewers for their thoughtful comments and feedback. Finally, we would like to thank Dr. Alexis Elder for her contributions regarding the ethical considerations of this work. Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 16th International Workshop on Semantic Evaluation, SemEval 2022 ; Conference date: 14-07-2022 Through 15-07-2022",

year = "2022",

language = "English (US)",

series = "SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop",

publisher = "Association for Computational Linguistics (ACL)",

pages = "1062--1066",

editor = "Guy Emerson and Natalie Schluter and Gabriel Stanovsky and Ritesh Kumar and Alexis Palmer and Nathan Schneider and Siddharth Singh and Shyam Ratan",

booktitle = "SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop",

}

TY - GEN

T1 - DuluthNLP at SemEval-2022 Task 7

T2 - 16th International Workshop on Semantic Evaluation, SemEval 2022

AU - Akrah, Samuel

AU - Pedersen, Ted

N1 - Funding Information: The authors would like to thank the organizers for the opportunity to participate in SemEval-2021 Task 7. We are also grateful to the three anonymous reviewers for their thoughtful comments and feedback. Finally, we would like to thank Dr. Alexis Elder for her contributions regarding the ethical considerations of this work. Publisher Copyright: © 2022 Association for Computational Linguistics.

PY - 2022

Y1 - 2022

N2 - This paper describes the DuluthNLP system that participated in Task 7 of SemEval-2022 on Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts. Given an instructional text with an omitted token, the task requires models to classify or rank the plausibility of potential fillers. To solve the task, we fine-tuned the models BERT, RoBERTa, and ELECTRA on training data where potential fillers are rated for plausibility. This is a challenging problem, as shown by BERT-based models achieving accuracy less than 45%. However, our ELECTRA model with tuned class weights on CrossEntropyLoss achieves an accuracy of 53.3% on the official evaluation test data, which ranks 6 out of the 8 total submissions for Subtask A.

AB - This paper describes the DuluthNLP system that participated in Task 7 of SemEval-2022 on Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts. Given an instructional text with an omitted token, the task requires models to classify or rank the plausibility of potential fillers. To solve the task, we fine-tuned the models BERT, RoBERTa, and ELECTRA on training data where potential fillers are rated for plausibility. This is a challenging problem, as shown by BERT-based models achieving accuracy less than 45%. However, our ELECTRA model with tuned class weights on CrossEntropyLoss achieves an accuracy of 53.3% on the official evaluation test data, which ranks 6 out of the 8 total submissions for Subtask A.

UR - http://www.scopus.com/inward/record.url?scp=85137589846&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85137589846&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85137589846

T3 - SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop

SP - 1062

EP - 1066

BT - SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop

A2 - Emerson, Guy

A2 - Schluter, Natalie

A2 - Stanovsky, Gabriel

A2 - Kumar, Ritesh

A2 - Palmer, Alexis

A2 - Schneider, Nathan

A2 - Singh, Siddharth

A2 - Ratan, Shyam

PB - Association for Computational Linguistics (ACL)

Y2 - 14 July 2022 through 15 July 2022

ER -

DuluthNLP at SemEval-2022 Task 7: Classifying Plausible Alternatives with Pre-trained ELECTRA

Abstract

Publication series

Conference

Bibliographical note

Find It

Other files and links

Fingerprint

Cite this