We propose a novel method for approximate hardware implementation of univariate math functions with significantly fewer hardware resources compared to previous approaches. Examples of such functions include exp(x) and the activation function GELU(x), both used in transformer networks, gamma(x), which is used in image processing, and other functions such as tanh(x), cosh(x), sq(x), and sqrt(x). The method builds on previous works on hybrid binary-unary computing. The novelty in our approach is that we break a function into a number of sub-functions such that implementing each sub-function becomes cheap, and converting the output of the sub-functions to binary becomes almost trivial. Our method also uses self-similarity in functions to further reduce the cost. We compare our method to the conventional binary, previous stochastic computing, and hybrid binary-unary methods on several functions at 8-, 12-, and 16-bit resolutions. While preserving high accuracy, our method outperforms previous works in terms of hardware cost, e.g., tolerating less than 0.01 mean absolute error, our method reduces the (area x latency) cost on average by 5, 7, and 2 orders of magnitude, compared to the conventional binary, stochastic computing, and hybrid binary-unary methods, respectively. Ultimately, we demonstrate the potential benefits of our method for natural language processing and image processing applications. We deploy our method to implement major blocks in an encoding layer of BERT language model, and also the Roberts Cross edge detection algorithm. Both include non-linear functions.
|Original language||English (US)|
|Title of host publication||FPGA 2023 - Proceedings of the 2023 ACM/SIGDA International Symposium on Field Programmable Gate Arrays|
|Publisher||Association for Computing Machinery, Inc|
|Number of pages||11|
|State||Published - Feb 12 2023|
|Event||31st ACM International Symposium on Field-Programmable Gate Arrays, FPGA 2023 - Monterey, United States|
Duration: Feb 12 2023 → Feb 14 2023
|Name||Proceedings of the 2023 ACM/SIGDA International Symposium on Field Programmable Gate Arrays|
|Conference||31st ACM International Symposium on Field-Programmable Gate Arrays, FPGA 2023|
|Period||2/12/23 → 2/14/23|
Bibliographical noteFunding Information:
This material is based upon work supported in part by the National Science Foundation under grant number PFI-TT 2016390, and by Cisco Systems, Inc. under grant number 1044678.
© 2023 ACM.
- BERT language model
- approximate computing
- hardware accelerators
- image processing
- stochastic computing
- unary computing