Ranking the parameters of deep neural networks using the fisher information

Ming Tu; Visar Berisha; Martin Woolf; Jae Sun Seo; Yu Cao

doi:10.1109/ICASSP.2016.7472157

Ranking the parameters of deep neural networks using the fisher information

Ming Tu
, Visar Berisha
, Martin Woolf
, Jae Sun Seo
, Yu Cao

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

23 Scopus citations

Abstract

The large number of parameters in deep neural networks (DNNs) often makes them prohibitive for low-power devices, such as field-programmable gate arrays (FPGA). In this paper, we propose a method to determine the relative importance of all network parameters by measuring the amount of information that the network output carries about each of the parameters - the Fisher Information. Based on the importance ranking, we design a complexity reduction scheme that discards unimportant parameters and assigns more quantization bits to more important parameters. For evaluation, we construct a deep autoencoder and learn a non-linear dimensionality reduction scheme for accelerometer data measuring the gait of individuals with Parkinson's disease. Experimental results confirm that the proposed ranking method can help reduce the complexity of the network with minimal impact on performance.

Original language	English (US)
Title of host publication	2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	2647-2651
Number of pages	5
ISBN (Electronic)	9781479999880
DOIs	https://doi.org/10.1109/ICASSP.2016.7472157
State	Published - May 18 2016
Externally published	Yes
Event	41st IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Shanghai, China Duration: Mar 20 2016 → Mar 25 2016

Publication series

Name	ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
Volume	2016-May
ISSN (Print)	1520-6149

Other

Other	41st IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016
Country/Territory	China
City	Shanghai
Period	3/20/16 → 3/25/16

Bibliographical note

Publisher Copyright:
© 2016 IEEE.

Keywords

FPGA
Fisher Information
complexity reduction
deep neural networks
pruning
quantization

Publisher link

10.1109/ICASSP.2016.7472157

Find It

Find It at U of M Libraries

Cite this

Tu, M., Berisha, V., Woolf, M., Seo, J. S., & Cao, Y. (2016). Ranking the parameters of deep neural networks using the fisher information. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings (pp. 2647-2651). Article 7472157 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings; Vol. 2016-May). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICASSP.2016.7472157

Ranking the parameters of deep neural networks using the fisher information. / Tu, Ming; Berisha, Visar; Woolf, Martin et al.
2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2016. p. 2647-2651 7472157 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings; Vol. 2016-May).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Tu, M, Berisha, V, Woolf, M, Seo, JS & Cao, Y 2016, Ranking the parameters of deep neural networks using the fisher information. in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings., 7472157, ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, vol. 2016-May, Institute of Electrical and Electronics Engineers Inc., pp. 2647-2651, 41st IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016, Shanghai, China, 3/20/16. https://doi.org/10.1109/ICASSP.2016.7472157

Tu M, Berisha V, Woolf M, Seo JS, Cao Y. Ranking the parameters of deep neural networks using the fisher information. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2016. p. 2647-2651. 7472157. (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings). doi: 10.1109/ICASSP.2016.7472157

Tu, Ming ; Berisha, Visar ; Woolf, Martin et al. / Ranking the parameters of deep neural networks using the fisher information. 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2016. pp. 2647-2651 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings).

@inproceedings{d4b366632ec847c093a40b8ed12b735a,

title = "Ranking the parameters of deep neural networks using the fisher information",

abstract = "The large number of parameters in deep neural networks (DNNs) often makes them prohibitive for low-power devices, such as field-programmable gate arrays (FPGA). In this paper, we propose a method to determine the relative importance of all network parameters by measuring the amount of information that the network output carries about each of the parameters - the Fisher Information. Based on the importance ranking, we design a complexity reduction scheme that discards unimportant parameters and assigns more quantization bits to more important parameters. For evaluation, we construct a deep autoencoder and learn a non-linear dimensionality reduction scheme for accelerometer data measuring the gait of individuals with Parkinson's disease. Experimental results confirm that the proposed ranking method can help reduce the complexity of the network with minimal impact on performance.",

keywords = "FPGA, Fisher Information, complexity reduction, deep neural networks, pruning, quantization",

author = "Ming Tu and Visar Berisha and Martin Woolf and Seo, \{Jae Sun\} and Yu Cao",

note = "Publisher Copyright: {\textcopyright} 2016 IEEE.; 41st IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 ; Conference date: 20-03-2016 Through 25-03-2016",

year = "2016",

month = may,

day = "18",

doi = "10.1109/ICASSP.2016.7472157",

language = "English (US)",

series = "ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "2647--2651",

booktitle = "2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings",

}

TY - GEN

T1 - Ranking the parameters of deep neural networks using the fisher information

AU - Tu, Ming

AU - Berisha, Visar

AU - Woolf, Martin

AU - Seo, Jae Sun

AU - Cao, Yu

PY - 2016/5/18

Y1 - 2016/5/18

N2 - The large number of parameters in deep neural networks (DNNs) often makes them prohibitive for low-power devices, such as field-programmable gate arrays (FPGA). In this paper, we propose a method to determine the relative importance of all network parameters by measuring the amount of information that the network output carries about each of the parameters - the Fisher Information. Based on the importance ranking, we design a complexity reduction scheme that discards unimportant parameters and assigns more quantization bits to more important parameters. For evaluation, we construct a deep autoencoder and learn a non-linear dimensionality reduction scheme for accelerometer data measuring the gait of individuals with Parkinson's disease. Experimental results confirm that the proposed ranking method can help reduce the complexity of the network with minimal impact on performance.

AB - The large number of parameters in deep neural networks (DNNs) often makes them prohibitive for low-power devices, such as field-programmable gate arrays (FPGA). In this paper, we propose a method to determine the relative importance of all network parameters by measuring the amount of information that the network output carries about each of the parameters - the Fisher Information. Based on the importance ranking, we design a complexity reduction scheme that discards unimportant parameters and assigns more quantization bits to more important parameters. For evaluation, we construct a deep autoencoder and learn a non-linear dimensionality reduction scheme for accelerometer data measuring the gait of individuals with Parkinson's disease. Experimental results confirm that the proposed ranking method can help reduce the complexity of the network with minimal impact on performance.

KW - FPGA

KW - Fisher Information

KW - complexity reduction

KW - deep neural networks

KW - pruning

KW - quantization

UR - https://www.scopus.com/pages/publications/84973402931

UR - https://www.scopus.com/pages/publications/84973402931#tab=citedBy

U2 - 10.1109/ICASSP.2016.7472157

DO - 10.1109/ICASSP.2016.7472157

M3 - Conference contribution

AN - SCOPUS:84973402931

T3 - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

SP - 2647

EP - 2651

BT - 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 41st IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016

Y2 - 20 March 2016 through 25 March 2016

ER -

Ranking the parameters of deep neural networks using the fisher information

Abstract

Publication series

Other

Bibliographical note

Keywords

Publisher link

Find It

Other files and links

Fingerprint

Cite this