Abstract
Although Transformer-based language representations achieve state-of-the-art accuracy on various natural language processing (NLP) tasks, the large model size has been challenging the resource constrained computing platforms. Weight pruning, as a popular and effective technique in reducing the number of weight parameters and accelerating the Transformer, has been investigated on GPUs. However, the Transformer acceleration using weight pruning on field-programmable gate array (FPGAs) remains unexplored. This paper investigates the column balanced block-wise pruning on Transformer and designs an FPGA acceleration engine to customize the balanced blockwise matrix multiplication. We implement the Transformer model with proper hardware scheduling, and the experiments show that the Transformer inference on FPGA achieves 10.35 ms latency with the batch size of 32, which is $10.96 \times$ speed up comparing to CPU platform and $2.08 \times$ speed up comparing to GPU platform.
Original language | English (US) |
---|---|
Title of host publication | Proceedings of the 22nd International Symposium on Quality Electronic Design, ISQED 2021 |
Publisher | IEEE Computer Society |
Pages | 142-148 |
Number of pages | 7 |
ISBN (Electronic) | 9781728176413 |
DOIs | |
State | Published - Apr 7 2021 |
Externally published | Yes |
Event | 22nd International Symposium on Quality Electronic Design, ISQED 2021 - Santa Clara, United States Duration: Apr 7 2021 → Apr 9 2021 |
Publication series
Name | Proceedings - International Symposium on Quality Electronic Design, ISQED |
---|---|
Volume | 2021-April |
ISSN (Print) | 1948-3287 |
ISSN (Electronic) | 1948-3295 |
Conference
Conference | 22nd International Symposium on Quality Electronic Design, ISQED 2021 |
---|---|
Country/Territory | United States |
City | Santa Clara |
Period | 4/7/21 → 4/9/21 |
Bibliographical note
Publisher Copyright:© 2021 IEEE.
Keywords
- Acceleration
- Deep learning
- FPGA
- Pruning
- Transformer