On The Energy Statistics of Feature Maps in Pruning of Neural Networks with Skip-Connections

Mohammadreza Soltani; Suya Wu; Yuerong Li; Jie Ding; Vahid Tarokh; Mohammadreza Soltani; Suya Wu

doi:10.1109/DCC52660.2022.00093

On The Energy Statistics of Feature Maps in Pruning of Neural Networks with Skip-Connections

Mohammadreza Soltani, Suya Wu, Yuerong Li, Jie Ding, Vahid Tarokh, Mohammadreza Soltani, Suya Wu

Statistics (Twin Cities)

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We propose a new structured pruning framework for compressing Deep Neural Networks (DNNs) with skip-connections, based on measuring the statistical dependency of hidden layers and predicted outputs. The dependence measure defined by the energy statistics of hidden layers serves as a model-free measure of information between the feature maps and the output of the network. The estimated dependence measure is subsequently used to prune a collection of redundant and uninformative layers. Extensive numerical experiments on various architectures show the efficacy of the proposed pruning approach with competitive performance to state-of-the-art methods.

Original language	English (US)
Title of host publication	Proceedings - DCC 2022
Subtitle of host publication	2022 Data Compression Conference
Editors	Ali Bilgin, Michael W. Marcellin, Joan Serra-Sagrista, James A. Storer
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	482
Number of pages	1
ISBN (Electronic)	9781665478939
DOIs	https://doi.org/10.1109/DCC52660.2022.00093
State	Published - 2022
Event	2022 Data Compression Conference, DCC 2022 - Snowbird, United States Duration: Mar 22 2022 → Mar 25 2022

Publication series

Name	2022 Data Compression Conference (DCC)

Conference

Conference	2022 Data Compression Conference, DCC 2022
Country/Territory	United States
City	Snowbird
Period	3/22/22 → 3/25/22

Bibliographical note

Funding Information:
This work has been supported in part by the Army Research Office grant No. W911NF-15-1-0479.

Publisher Copyright:
© 2022 IEEE.

Publisher link

10.1109/DCC52660.2022.00093

Find It

Find It at U of M Libraries

Cite this

Soltani, M., Wu, S., Li, Y., Ding, J., Tarokh, V., Soltani, M., & Wu, S. (2022). On The Energy Statistics of Feature Maps in Pruning of Neural Networks with Skip-Connections. In A. Bilgin, M. W. Marcellin, J. Serra-Sagrista, & J. A. Storer (Eds.), Proceedings - DCC 2022: 2022 Data Compression Conference (pp. 482). (2022 Data Compression Conference (DCC)). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/DCC52660.2022.00093

On The Energy Statistics of Feature Maps in Pruning of Neural Networks with Skip-Connections. / Soltani, Mohammadreza; Wu, Suya; Li, Yuerong et al.
Proceedings - DCC 2022: 2022 Data Compression Conference. ed. / Ali Bilgin; Michael W. Marcellin; Joan Serra-Sagrista; James A. Storer. Institute of Electrical and Electronics Engineers Inc., 2022. p. 482 (2022 Data Compression Conference (DCC)).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Soltani, M, Wu, S, Li, Y, Ding, J, Tarokh, V, Soltani, M & Wu, S 2022, On The Energy Statistics of Feature Maps in Pruning of Neural Networks with Skip-Connections. in A Bilgin, MW Marcellin, J Serra-Sagrista & JA Storer (eds), Proceedings - DCC 2022: 2022 Data Compression Conference. 2022 Data Compression Conference (DCC), Institute of Electrical and Electronics Engineers Inc., pp. 482, 2022 Data Compression Conference, DCC 2022, Snowbird, United States, 3/22/22. https://doi.org/10.1109/DCC52660.2022.00093

Soltani M, Wu S, Li Y, Ding J, Tarokh V, Soltani M et al. On The Energy Statistics of Feature Maps in Pruning of Neural Networks with Skip-Connections. In Bilgin A, Marcellin MW, Serra-Sagrista J, Storer JA, editors, Proceedings - DCC 2022: 2022 Data Compression Conference. Institute of Electrical and Electronics Engineers Inc. 2022. p. 482. (2022 Data Compression Conference (DCC)). doi: 10.1109/DCC52660.2022.00093

Soltani, Mohammadreza ; Wu, Suya ; Li, Yuerong et al. / On The Energy Statistics of Feature Maps in Pruning of Neural Networks with Skip-Connections. Proceedings - DCC 2022: 2022 Data Compression Conference. editor / Ali Bilgin ; Michael W. Marcellin ; Joan Serra-Sagrista ; James A. Storer. Institute of Electrical and Electronics Engineers Inc., 2022. pp. 482 (2022 Data Compression Conference (DCC)).

@inproceedings{eb32e72a980d4516bf7a5fc59e8b6ad7,

title = "On The Energy Statistics of Feature Maps in Pruning of Neural Networks with Skip-Connections",

abstract = "We propose a new structured pruning framework for compressing Deep Neural Networks (DNNs) with skip-connections, based on measuring the statistical dependency of hidden layers and predicted outputs. The dependence measure defined by the energy statistics of hidden layers serves as a model-free measure of information between the feature maps and the output of the network. The estimated dependence measure is subsequently used to prune a collection of redundant and uninformative layers. Extensive numerical experiments on various architectures show the efficacy of the proposed pruning approach with competitive performance to state-of-the-art methods.",

author = "Mohammadreza Soltani and Suya Wu and Yuerong Li and Jie Ding and Vahid Tarokh and Mohammadreza Soltani and Suya Wu",

note = "Funding Information: This work has been supported in part by the Army Research Office grant No. W911NF-15-1-0479. Publisher Copyright: {\textcopyright} 2022 IEEE.; 2022 Data Compression Conference, DCC 2022 ; Conference date: 22-03-2022 Through 25-03-2022",

year = "2022",

doi = "10.1109/DCC52660.2022.00093",

language = "English (US)",

series = "2022 Data Compression Conference (DCC)",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "482",

editor = "Ali Bilgin and Marcellin, {Michael W.} and Joan Serra-Sagrista and Storer, {James A.}",

booktitle = "Proceedings - DCC 2022",

}

TY - GEN

T1 - On The Energy Statistics of Feature Maps in Pruning of Neural Networks with Skip-Connections

AU - Soltani, Mohammadreza

AU - Wu, Suya

AU - Li, Yuerong

AU - Ding, Jie

AU - Tarokh, Vahid

AU - Soltani, Mohammadreza

AU - Wu, Suya

PY - 2022

Y1 - 2022

N2 - We propose a new structured pruning framework for compressing Deep Neural Networks (DNNs) with skip-connections, based on measuring the statistical dependency of hidden layers and predicted outputs. The dependence measure defined by the energy statistics of hidden layers serves as a model-free measure of information between the feature maps and the output of the network. The estimated dependence measure is subsequently used to prune a collection of redundant and uninformative layers. Extensive numerical experiments on various architectures show the efficacy of the proposed pruning approach with competitive performance to state-of-the-art methods.

AB - We propose a new structured pruning framework for compressing Deep Neural Networks (DNNs) with skip-connections, based on measuring the statistical dependency of hidden layers and predicted outputs. The dependence measure defined by the energy statistics of hidden layers serves as a model-free measure of information between the feature maps and the output of the network. The estimated dependence measure is subsequently used to prune a collection of redundant and uninformative layers. Extensive numerical experiments on various architectures show the efficacy of the proposed pruning approach with competitive performance to state-of-the-art methods.

UR - http://www.scopus.com/inward/record.url?scp=85134378424&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85134378424&partnerID=8YFLogxK

U2 - 10.1109/DCC52660.2022.00093

DO - 10.1109/DCC52660.2022.00093

M3 - Conference contribution

AN - SCOPUS:85134378424

T3 - 2022 Data Compression Conference (DCC)

SP - 482

BT - Proceedings - DCC 2022

A2 - Bilgin, Ali

A2 - Marcellin, Michael W.

A2 - Serra-Sagrista, Joan

A2 - Storer, James A.

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2022 Data Compression Conference, DCC 2022

Y2 - 22 March 2022 through 25 March 2022

ER -

On The Energy Statistics of Feature Maps in Pruning of Neural Networks with Skip-Connections

Abstract

Publication series

Conference

Bibliographical note

Publisher link

Find It

Other files and links

Fingerprint

Cite this