Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm

Shaoyi Huang; Dongkuan Xu; Ian En Hsu Yen; Yijue Wang; Sung En Chang; Bingbing Li; Shiyang Chen; Mimi Xie; Sanguthevar Rajasekaran; Hang Liu; Caiwen Ding

doi:10.18653/v1/2022.acl-long.16

Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm

Shaoyi Huang
, Dongkuan Xu
, Ian En Hsu Yen
, Yijue Wang
, Sung En Chang
, Bingbing Li
, Shiyang Chen
, Mimi Xie
, Sanguthevar Rajasekaran
, Hang Liu
, Caiwen Ding

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

12 Scopus citations

Abstract

Conventional wisdom in pruning Transformer-based language models is that pruning reduces the model expressiveness and thus is more likely to underfit rather than overfit. However, under the trending pretrain-and-finetune paradigm, we postulate a counter-traditional hypothesis, that is: pruning increases the risk of overfitting when performed at the fine-tuning phase. In this paper, we aim to address the overfitting problem and improve pruning performance via progressive knowledge distillation with error-bound properties. We show for the first time that reducing the risk of overfitting can help the effectiveness of pruning under the pretrain-and-finetune paradigm. Ablation studies and experiments on the GLUE benchmark show that our method outperforms the leading competitors across different tasks.

Original language	English (US)
Title of host publication	ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)
Editors	Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Publisher	Association for Computational Linguistics (ACL)
Pages	190-200
Number of pages	11
ISBN (Electronic)	9781955917216
DOIs	https://doi.org/10.18653/v1/2022.acl-long.16
State	Published - 2022
Externally published	Yes
Event	60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 - Dublin, Ireland Duration: May 22 2022 → May 27 2022

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
Volume	1
ISSN (Print)	0736-587X

Conference

Conference	60th Annual Meeting of the Association for Computational Linguistics, ACL 2022
Country/Territory	Ireland
City	Dublin
Period	5/22/22 → 5/27/22

Bibliographical note

Publisher Copyright:
© 2022 Association for Computational Linguistics.

Publisher link

10.18653/v1/2022.acl-long.16

Find It

Find It at U of M Libraries

Cite this

Huang, S., Xu, D., Yen, I. E. H., Wang, Y., Chang, S. E., Li, B., Chen, S., Xie, M., Rajasekaran, S., Liu, H., & Ding, C. (2022). Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm. In S. Muresan, P. Nakov, & A. Villavicencio (Eds.), ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers) (pp. 190-200). (Proceedings of the Annual Meeting of the Association for Computational Linguistics; Vol. 1). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2022.acl-long.16

Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm. / Huang, Shaoyi; Xu, Dongkuan; Yen, Ian En Hsu et al.
ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). ed. / Smaranda Muresan; Preslav Nakov; Aline Villavicencio. Association for Computational Linguistics (ACL), 2022. p. 190-200 (Proceedings of the Annual Meeting of the Association for Computational Linguistics; Vol. 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Huang, S, Xu, D, Yen, IEH, Wang, Y, Chang, SE, Li, B, Chen, S, Xie, M, Rajasekaran, S, Liu, H & Ding, C 2022, Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm. in S Muresan, P Nakov & A Villavicencio (eds), ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). Proceedings of the Annual Meeting of the Association for Computational Linguistics, vol. 1, Association for Computational Linguistics (ACL), pp. 190-200, 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022, Dublin, Ireland, 5/22/22. https://doi.org/10.18653/v1/2022.acl-long.16

Huang S, Xu D, Yen IEH, Wang Y, Chang SE, Li B et al. Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm. In Muresan S, Nakov P, Villavicencio A, editors, ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). Association for Computational Linguistics (ACL). 2022. p. 190-200. (Proceedings of the Annual Meeting of the Association for Computational Linguistics). doi: 10.18653/v1/2022.acl-long.16

Huang, Shaoyi ; Xu, Dongkuan ; Yen, Ian En Hsu et al. / Sparse Progressive Distillation : Resolving Overfitting under Pretrain-and-Finetune Paradigm. ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). editor / Smaranda Muresan ; Preslav Nakov ; Aline Villavicencio. Association for Computational Linguistics (ACL), 2022. pp. 190-200 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{b94e91306d1648a18f8c88bd4e5f4bdf,

title = "Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm",

abstract = "Conventional wisdom in pruning Transformer-based language models is that pruning reduces the model expressiveness and thus is more likely to underfit rather than overfit. However, under the trending pretrain-and-finetune paradigm, we postulate a counter-traditional hypothesis, that is: pruning increases the risk of overfitting when performed at the fine-tuning phase. In this paper, we aim to address the overfitting problem and improve pruning performance via progressive knowledge distillation with error-bound properties. We show for the first time that reducing the risk of overfitting can help the effectiveness of pruning under the pretrain-and-finetune paradigm. Ablation studies and experiments on the GLUE benchmark show that our method outperforms the leading competitors across different tasks.",

author = "Shaoyi Huang and Dongkuan Xu and Yen, \{Ian En Hsu\} and Yijue Wang and Chang, \{Sung En\} and Bingbing Li and Shiyang Chen and Mimi Xie and Sanguthevar Rajasekaran and Hang Liu and Caiwen Ding",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 ; Conference date: 22-05-2022 Through 27-05-2022",

year = "2022",

doi = "10.18653/v1/2022.acl-long.16",

language = "English (US)",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

pages = "190--200",

editor = "Smaranda Muresan and Preslav Nakov and Aline Villavicencio",

booktitle = "ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)",

}

TY - GEN

T1 - Sparse Progressive Distillation

T2 - 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022

AU - Huang, Shaoyi

AU - Xu, Dongkuan

AU - Yen, Ian En Hsu

AU - Wang, Yijue

AU - Chang, Sung En

AU - Li, Bingbing

AU - Chen, Shiyang

AU - Xie, Mimi

AU - Rajasekaran, Sanguthevar

AU - Liu, Hang

AU - Ding, Caiwen

PY - 2022

Y1 - 2022

N2 - Conventional wisdom in pruning Transformer-based language models is that pruning reduces the model expressiveness and thus is more likely to underfit rather than overfit. However, under the trending pretrain-and-finetune paradigm, we postulate a counter-traditional hypothesis, that is: pruning increases the risk of overfitting when performed at the fine-tuning phase. In this paper, we aim to address the overfitting problem and improve pruning performance via progressive knowledge distillation with error-bound properties. We show for the first time that reducing the risk of overfitting can help the effectiveness of pruning under the pretrain-and-finetune paradigm. Ablation studies and experiments on the GLUE benchmark show that our method outperforms the leading competitors across different tasks.

AB - Conventional wisdom in pruning Transformer-based language models is that pruning reduces the model expressiveness and thus is more likely to underfit rather than overfit. However, under the trending pretrain-and-finetune paradigm, we postulate a counter-traditional hypothesis, that is: pruning increases the risk of overfitting when performed at the fine-tuning phase. In this paper, we aim to address the overfitting problem and improve pruning performance via progressive knowledge distillation with error-bound properties. We show for the first time that reducing the risk of overfitting can help the effectiveness of pruning under the pretrain-and-finetune paradigm. Ablation studies and experiments on the GLUE benchmark show that our method outperforms the leading competitors across different tasks.

UR - https://www.scopus.com/pages/publications/85138618135

UR - https://www.scopus.com/pages/publications/85138618135#tab=citedBy

U2 - 10.18653/v1/2022.acl-long.16

DO - 10.18653/v1/2022.acl-long.16

M3 - Conference contribution

AN - SCOPUS:85138618135

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 190

EP - 200

BT - ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)

A2 - Muresan, Smaranda

A2 - Nakov, Preslav

A2 - Villavicencio, Aline

PB - Association for Computational Linguistics (ACL)

Y2 - 22 May 2022 through 27 May 2022

ER -

Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm

Abstract

Publication series

Conference

Bibliographical note

Publisher link

Find It

Other files and links

Fingerprint

Cite this