You Already Have It: A Generator-Free Low-Precision DNN Training Framework Using Stochastic Rounding

Geng Yuan; Sung En Chang; Qing Jin; Alec Lu; Yanyu Li; Yushu Wu; Zhenglun Kong; Yanyue Xie; Peiyan Dong; Minghai Qin; Xiaolong Ma; Xulong Tang; Zhenman Fang; Yanzhi Wang

doi:10.1007/978-3-031-19775-8_3

You Already Have It: A Generator-Free Low-Precision DNN Training Framework Using Stochastic Rounding

Geng Yuan
, Sung En Chang
, Qing Jin
, Alec Lu
, Yanyu Li
, Yushu Wu
, Zhenglun Kong
, Yanyue Xie
, Peiyan Dong
, Minghai Qin
, Xiaolong Ma
, Xulong Tang
, Zhenman Fang
, Yanzhi Wang

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

6 Scopus citations

Abstract

Stochastic rounding is a critical technique used in low-precision deep neural networks (DNNs) training to ensure good model accuracy. However, it requires a large number of random numbers generated on the fly. This is not a trivial task on the hardware platforms such as FPGA and ASIC. The widely used solution is to introduce random number generators with extra hardware costs. In this paper, we innovatively propose to employ the stochastic property of DNN training process itself and directly extract random numbers from DNNs in a self-sufficient manner. We propose different methods to obtain random numbers from different sources in neural networks and a generator-free framework is proposed for low-precision DNN training on a variety of deep learning tasks. Moreover, we evaluate the quality of the extracted random numbers and find that high-quality random numbers widely exist in DNNs, while their quality can even pass the NIST test suite.

Original language	English (US)
Title of host publication	Computer Vision – ECCV 2022 - 17th European Conference, Proceedings
Editors	Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	34-51
Number of pages	18
ISBN (Print)	9783031197741
DOIs	https://doi.org/10.1007/978-3-031-19775-8_3
State	Published - 2022
Externally published	Yes
Event	17th European Conference on Computer Vision, ECCV 2022 - Tel Aviv, Israel Duration: Oct 23 2022 → Oct 27 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13672 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	17th European Conference on Computer Vision, ECCV 2022
Country/Territory	Israel
City	Tel Aviv
Period	10/23/22 → 10/27/22

Bibliographical note

Publisher Copyright:
© 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Keywords

Efficient training
Quantization
Stochastic rounding

Publisher link

10.1007/978-3-031-19775-8_3

Find It

Find It at U of M Libraries

Cite this

Yuan, G., Chang, S. E., Jin, Q., Lu, A., Li, Y., Wu, Y., Kong, Z., Xie, Y., Dong, P., Qin, M., Ma, X., Tang, X., Fang, Z., & Wang, Y. (2022). You Already Have It: A Generator-Free Low-Precision DNN Training Framework Using Stochastic Rounding. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, & T. Hassner (Eds.), Computer Vision – ECCV 2022 - 17th European Conference, Proceedings (pp. 34-51). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13672 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-19775-8_3

You Already Have It: A Generator-Free Low-Precision DNN Training Framework Using Stochastic Rounding. / Yuan, Geng; Chang, Sung En; Jin, Qing et al.
Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. ed. / Shai Avidan; Gabriel Brostow; Moustapha Cissé; Giovanni Maria Farinella; Tal Hassner. Springer Science and Business Media Deutschland GmbH, 2022. p. 34-51 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13672 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Yuan, G, Chang, SE, Jin, Q, Lu, A, Li, Y, Wu, Y, Kong, Z, Xie, Y, Dong, P, Qin, M, Ma, X, Tang, X, Fang, Z & Wang, Y 2022, You Already Have It: A Generator-Free Low-Precision DNN Training Framework Using Stochastic Rounding. in S Avidan, G Brostow, M Cissé, GM Farinella & T Hassner (eds), Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13672 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 34-51, 17th European Conference on Computer Vision, ECCV 2022, Tel Aviv, Israel, 10/23/22. https://doi.org/10.1007/978-3-031-19775-8_3

Yuan G, Chang SE, Jin Q, Lu A, Li Y, Wu Y et al. You Already Have It: A Generator-Free Low-Precision DNN Training Framework Using Stochastic Rounding. In Avidan S, Brostow G, Cissé M, Farinella GM, Hassner T, editors, Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. Springer Science and Business Media Deutschland GmbH. 2022. p. 34-51. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-19775-8_3

Yuan, Geng ; Chang, Sung En ; Jin, Qing et al. / You Already Have It : A Generator-Free Low-Precision DNN Training Framework Using Stochastic Rounding. Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. editor / Shai Avidan ; Gabriel Brostow ; Moustapha Cissé ; Giovanni Maria Farinella ; Tal Hassner. Springer Science and Business Media Deutschland GmbH, 2022. pp. 34-51 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{fc33dbe1edf545e5b489f0c4e6c09532,

title = "You Already Have It: A Generator-Free Low-Precision DNN Training Framework Using Stochastic Rounding",

abstract = "Stochastic rounding is a critical technique used in low-precision deep neural networks (DNNs) training to ensure good model accuracy. However, it requires a large number of random numbers generated on the fly. This is not a trivial task on the hardware platforms such as FPGA and ASIC. The widely used solution is to introduce random number generators with extra hardware costs. In this paper, we innovatively propose to employ the stochastic property of DNN training process itself and directly extract random numbers from DNNs in a self-sufficient manner. We propose different methods to obtain random numbers from different sources in neural networks and a generator-free framework is proposed for low-precision DNN training on a variety of deep learning tasks. Moreover, we evaluate the quality of the extracted random numbers and find that high-quality random numbers widely exist in DNNs, while their quality can even pass the NIST test suite.",

keywords = "Efficient training, Quantization, Stochastic rounding",

author = "Geng Yuan and Chang, \{Sung En\} and Qing Jin and Alec Lu and Yanyu Li and Yushu Wu and Zhenglun Kong and Yanyue Xie and Peiyan Dong and Minghai Qin and Xiaolong Ma and Xulong Tang and Zhenman Fang and Yanzhi Wang",

note = "Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 17th European Conference on Computer Vision, ECCV 2022 ; Conference date: 23-10-2022 Through 27-10-2022",

year = "2022",

doi = "10.1007/978-3-031-19775-8\_3",

language = "English (US)",

isbn = "9783031197741",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "34--51",

editor = "Shai Avidan and Gabriel Brostow and Moustapha Ciss{\'e} and Farinella, \{Giovanni Maria\} and Tal Hassner",

booktitle = "Computer Vision – ECCV 2022 - 17th European Conference, Proceedings",

address = "Germany",

}

TY - GEN

T1 - You Already Have It

T2 - 17th European Conference on Computer Vision, ECCV 2022

AU - Yuan, Geng

AU - Chang, Sung En

AU - Jin, Qing

AU - Lu, Alec

AU - Li, Yanyu

AU - Wu, Yushu

AU - Kong, Zhenglun

AU - Xie, Yanyue

AU - Dong, Peiyan

AU - Qin, Minghai

AU - Ma, Xiaolong

AU - Tang, Xulong

AU - Fang, Zhenman

AU - Wang, Yanzhi

PY - 2022

Y1 - 2022

N2 - Stochastic rounding is a critical technique used in low-precision deep neural networks (DNNs) training to ensure good model accuracy. However, it requires a large number of random numbers generated on the fly. This is not a trivial task on the hardware platforms such as FPGA and ASIC. The widely used solution is to introduce random number generators with extra hardware costs. In this paper, we innovatively propose to employ the stochastic property of DNN training process itself and directly extract random numbers from DNNs in a self-sufficient manner. We propose different methods to obtain random numbers from different sources in neural networks and a generator-free framework is proposed for low-precision DNN training on a variety of deep learning tasks. Moreover, we evaluate the quality of the extracted random numbers and find that high-quality random numbers widely exist in DNNs, while their quality can even pass the NIST test suite.

AB - Stochastic rounding is a critical technique used in low-precision deep neural networks (DNNs) training to ensure good model accuracy. However, it requires a large number of random numbers generated on the fly. This is not a trivial task on the hardware platforms such as FPGA and ASIC. The widely used solution is to introduce random number generators with extra hardware costs. In this paper, we innovatively propose to employ the stochastic property of DNN training process itself and directly extract random numbers from DNNs in a self-sufficient manner. We propose different methods to obtain random numbers from different sources in neural networks and a generator-free framework is proposed for low-precision DNN training on a variety of deep learning tasks. Moreover, we evaluate the quality of the extracted random numbers and find that high-quality random numbers widely exist in DNNs, while their quality can even pass the NIST test suite.

KW - Efficient training

KW - Quantization

KW - Stochastic rounding

UR - https://www.scopus.com/pages/publications/85142728811

UR - https://www.scopus.com/pages/publications/85142728811#tab=citedBy

U2 - 10.1007/978-3-031-19775-8_3

DO - 10.1007/978-3-031-19775-8_3

M3 - Conference contribution

AN - SCOPUS:85142728811

SN - 9783031197741

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 34

EP - 51

BT - Computer Vision – ECCV 2022 - 17th European Conference, Proceedings

A2 - Avidan, Shai

A2 - Brostow, Gabriel

A2 - Cissé, Moustapha

A2 - Farinella, Giovanni Maria

A2 - Hassner, Tal

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 23 October 2022 through 27 October 2022

ER -

You Already Have It: A Generator-Free Low-Precision DNN Training Framework Using Stochastic Rounding

Abstract

Publication series

Conference

Bibliographical note

Keywords

Publisher link

Find It

Other files and links

Fingerprint

Cite this