A dataset of peer reviews (PeerRead): Collection, insights and NLP applications

Dongyeop Kang; Waleed Ammar; Bhavana Dalvi Mishra; Madeleine Van Zuylen; Sebastian Kohlmeier; Eduard Hovy; Roy Schwartz

A dataset of peer reviews (PeerRead): Collection, insights and NLP applications

Dongyeop Kang, Waleed Ammar, Bhavana Dalvi Mishra, Madeleine Van Zuylen, Sebastian Kohlmeier, Eduard Hovy, Roy Schwartz

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Peer reviewing is a central component in the scientific publishing process. We present the first public dataset of scientific peer reviews available for research purposes (PeerRead v1),1 providing an opportunity to study this important artifact. The dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions in top-Tier venues including ACL, NIPS and ICLR. The dataset also includes 10.7K textual peer reviews written by experts for a subset of the papers. We describe the data collection process and report interesting observed phenomena in the peer reviews. We also propose two novel NLP tasks based on this dataset and provide simple baseline models. In the first task, we show that simple models can predict whether a paper is accepted with up to 21% error reduction compared to the majority baseline. In the second task, we predict the numerical scores of review aspects and show that simple models can outperform the mean baseline for aspects with high variance such as ?originality' and ?impact'.

Original language	English (US)
Title of host publication	Long Papers
Publisher	Association for Computational Linguistics (ACL)
Pages	1647-1661
Number of pages	15
ISBN (Electronic)	9781948087278
State	Published - 2018
Externally published	Yes
Event	2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018 - New Orleans, United States Duration: Jun 1 2018 → Jun 6 2018

Publication series

Name	NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference
Volume	1

Conference

Conference	2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018
Country/Territory	United States
City	New Orleans
Period	6/1/18 → 6/6/18

Bibliographical note

Publisher Copyright:
© 2018 The Association for Computational Linguistics.

Find It

Find It at U of M Libraries

Cite this

Kang, D., Ammar, W., Mishra, B. D., Van Zuylen, M., Kohlmeier, S., Hovy, E., & Schwartz, R. (2018). A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Long Papers (pp. 1647-1661). (NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference; Vol. 1). Association for Computational Linguistics (ACL).

A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. / Kang, Dongyeop; Ammar, Waleed; Mishra, Bhavana Dalvi et al.
Long Papers. Association for Computational Linguistics (ACL), 2018. p. 1647-1661 (NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference; Vol. 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Kang, D, Ammar, W, Mishra, BD, Van Zuylen, M, Kohlmeier, S, Hovy, E & Schwartz, R 2018, A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. in Long Papers. NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, vol. 1, Association for Computational Linguistics (ACL), pp. 1647-1661, 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018, New Orleans, United States, 6/1/18.

Kang D, Ammar W, Mishra BD, Van Zuylen M, Kohlmeier S, Hovy E et al. A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Long Papers. Association for Computational Linguistics (ACL). 2018. p. 1647-1661. (NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference).

Kang, Dongyeop ; Ammar, Waleed ; Mishra, Bhavana Dalvi et al. / A dataset of peer reviews (PeerRead) : Collection, insights and NLP applications. Long Papers. Association for Computational Linguistics (ACL), 2018. pp. 1647-1661 (NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference).

@inproceedings{6d5d6fa5b9c34710a3796b5f33ef2918,

title = "A dataset of peer reviews (PeerRead): Collection, insights and NLP applications",

abstract = "Peer reviewing is a central component in the scientific publishing process. We present the first public dataset of scientific peer reviews available for research purposes (PeerRead v1),1 providing an opportunity to study this important artifact. The dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions in top-Tier venues including ACL, NIPS and ICLR. The dataset also includes 10.7K textual peer reviews written by experts for a subset of the papers. We describe the data collection process and report interesting observed phenomena in the peer reviews. We also propose two novel NLP tasks based on this dataset and provide simple baseline models. In the first task, we show that simple models can predict whether a paper is accepted with up to 21% error reduction compared to the majority baseline. In the second task, we predict the numerical scores of review aspects and show that simple models can outperform the mean baseline for aspects with high variance such as ?originality' and ?impact'.",

author = "Dongyeop Kang and Waleed Ammar and Mishra, {Bhavana Dalvi} and {Van Zuylen}, Madeleine and Sebastian Kohlmeier and Eduard Hovy and Roy Schwartz",

note = "Publisher Copyright: {\textcopyright} 2018 The Association for Computational Linguistics.; 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018 ; Conference date: 01-06-2018 Through 06-06-2018",

year = "2018",

language = "English (US)",

series = "NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference",

publisher = "Association for Computational Linguistics (ACL)",

pages = "1647--1661",

booktitle = "Long Papers",

}

TY - GEN

T1 - A dataset of peer reviews (PeerRead)

T2 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018

AU - Kang, Dongyeop

AU - Ammar, Waleed

AU - Mishra, Bhavana Dalvi

AU - Van Zuylen, Madeleine

AU - Kohlmeier, Sebastian

AU - Hovy, Eduard

AU - Schwartz, Roy

PY - 2018

Y1 - 2018

N2 - Peer reviewing is a central component in the scientific publishing process. We present the first public dataset of scientific peer reviews available for research purposes (PeerRead v1),1 providing an opportunity to study this important artifact. The dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions in top-Tier venues including ACL, NIPS and ICLR. The dataset also includes 10.7K textual peer reviews written by experts for a subset of the papers. We describe the data collection process and report interesting observed phenomena in the peer reviews. We also propose two novel NLP tasks based on this dataset and provide simple baseline models. In the first task, we show that simple models can predict whether a paper is accepted with up to 21% error reduction compared to the majority baseline. In the second task, we predict the numerical scores of review aspects and show that simple models can outperform the mean baseline for aspects with high variance such as ?originality' and ?impact'.

AB - Peer reviewing is a central component in the scientific publishing process. We present the first public dataset of scientific peer reviews available for research purposes (PeerRead v1),1 providing an opportunity to study this important artifact. The dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions in top-Tier venues including ACL, NIPS and ICLR. The dataset also includes 10.7K textual peer reviews written by experts for a subset of the papers. We describe the data collection process and report interesting observed phenomena in the peer reviews. We also propose two novel NLP tasks based on this dataset and provide simple baseline models. In the first task, we show that simple models can predict whether a paper is accepted with up to 21% error reduction compared to the majority baseline. In the second task, we predict the numerical scores of review aspects and show that simple models can outperform the mean baseline for aspects with high variance such as ?originality' and ?impact'.

UR - http://www.scopus.com/inward/record.url?scp=85070968390&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85070968390&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85070968390

T3 - NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference

SP - 1647

EP - 1661

BT - Long Papers

PB - Association for Computational Linguistics (ACL)

Y2 - 1 June 2018 through 6 June 2018

ER -

A dataset of peer reviews (PeerRead): Collection, insights and NLP applications

Abstract

Publication series

Conference

Bibliographical note

Find It

Other files and links

Fingerprint

Cite this