Late Breaking Results: FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization

Mengshu Sun; Zhengang Li; Alec Lu; Haoyu Ma; Geng Yuan; Yanyue Xie; Hao Tang; Yanyu Li; Miriam Leeser; Zhangyang Wang; Xue Lin; Zhenman Fang

doi:10.1145/3489517.3530618

Late Breaking Results: FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization

Mengshu Sun
, Zhengang Li
, Alec Lu
, Haoyu Ma
, Geng Yuan
, Yanyue Xie
, Hao Tang
, Yanyu Li
, Miriam Leeser
, Zhangyang Wang
, Xue Lin
, Zhenman Fang

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

9 Scopus citations

Abstract

Vision transformers (ViTs) are emerging with significantly improved accuracy in computer vision tasks. However, their complex architecture and enormous computation/storage demand impose urgent needs for new hardware accelerator design methodology. This work proposes an FPGA-aware automatic ViT acceleration framework based on the proposed mixed-scheme quantization. To the best of our knowledge, this is the first FPGA-based ViT acceleration framework exploring model quantization. Compared with state-of-the-art ViT quantization work (algorithmic approach only without hardware acceleration), our quantization achieves 0.31% to 1.25% higher Top-1 accuracy under the same bit-width. Compared with the 32-bit floating-point baseline FPGA accelerator, our accelerator achieves around 5.6× improvement on the frame rate (i.e., 56.4 FPS vs. 10.0 FPS) with 0.83% accuracy drop for DeiT-base.

Original language	English (US)
Title of host publication	Proceedings of the 59th ACM/IEEE Design Automation Conference, DAC 2022
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1394-1395
Number of pages	2
ISBN (Electronic)	9781450391429
DOIs	https://doi.org/10.1145/3489517.3530618
State	Published - Jul 10 2022
Externally published	Yes
Event	59th ACM/IEEE Design Automation Conference, DAC 2022 - San Francisco, United States Duration: Jul 10 2022 → Jul 14 2022

Publication series

Name	Proceedings - Design Automation Conference
ISSN (Print)	0738-100X

Conference

Conference	59th ACM/IEEE Design Automation Conference, DAC 2022
Country/Territory	United States
City	San Francisco
Period	7/10/22 → 7/14/22

Bibliographical note

Publisher Copyright:
© 2022 ACM.

Publisher link

10.1145/3489517.3530618

Find It

Find It at U of M Libraries

Cite this

Sun, M., Li, Z., Lu, A., Ma, H., Yuan, G., Xie, Y., Tang, H., Li, Y., Leeser, M., Wang, Z., Lin, X., & Fang, Z. (2022). Late Breaking Results: FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization. In Proceedings of the 59th ACM/IEEE Design Automation Conference, DAC 2022 (pp. 1394-1395). (Proceedings - Design Automation Conference). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1145/3489517.3530618

Late Breaking Results: FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization. / Sun, Mengshu; Li, Zhengang; Lu, Alec et al.
Proceedings of the 59th ACM/IEEE Design Automation Conference, DAC 2022. Institute of Electrical and Electronics Engineers Inc., 2022. p. 1394-1395 (Proceedings - Design Automation Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Sun, M, Li, Z, Lu, A, Ma, H, Yuan, G, Xie, Y, Tang, H, Li, Y, Leeser, M, Wang, Z, Lin, X & Fang, Z 2022, Late Breaking Results: FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization. in Proceedings of the 59th ACM/IEEE Design Automation Conference, DAC 2022. Proceedings - Design Automation Conference, Institute of Electrical and Electronics Engineers Inc., pp. 1394-1395, 59th ACM/IEEE Design Automation Conference, DAC 2022, San Francisco, United States, 7/10/22. https://doi.org/10.1145/3489517.3530618

Sun M, Li Z, Lu A, Ma H, Yuan G, Xie Y et al. Late Breaking Results: FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization. In Proceedings of the 59th ACM/IEEE Design Automation Conference, DAC 2022. Institute of Electrical and Electronics Engineers Inc. 2022. p. 1394-1395. (Proceedings - Design Automation Conference). doi: 10.1145/3489517.3530618

@inproceedings{91cdc5e8b3f04c42b5f92ac8e0b0097b,

title = "Late Breaking Results: FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization",

abstract = "Vision transformers (ViTs) are emerging with significantly improved accuracy in computer vision tasks. However, their complex architecture and enormous computation/storage demand impose urgent needs for new hardware accelerator design methodology. This work proposes an FPGA-aware automatic ViT acceleration framework based on the proposed mixed-scheme quantization. To the best of our knowledge, this is the first FPGA-based ViT acceleration framework exploring model quantization. Compared with state-of-the-art ViT quantization work (algorithmic approach only without hardware acceleration), our quantization achieves 0.31\% to 1.25\% higher Top-1 accuracy under the same bit-width. Compared with the 32-bit floating-point baseline FPGA accelerator, our accelerator achieves around 5.6× improvement on the frame rate (i.e., 56.4 FPS vs. 10.0 FPS) with 0.83\% accuracy drop for DeiT-base.",

author = "Mengshu Sun and Zhengang Li and Alec Lu and Haoyu Ma and Geng Yuan and Yanyue Xie and Hao Tang and Yanyu Li and Miriam Leeser and Zhangyang Wang and Xue Lin and Zhenman Fang",

note = "Publisher Copyright: {\textcopyright} 2022 ACM.; 59th ACM/IEEE Design Automation Conference, DAC 2022 ; Conference date: 10-07-2022 Through 14-07-2022",

year = "2022",

month = jul,

day = "10",

doi = "10.1145/3489517.3530618",

language = "English (US)",

series = "Proceedings - Design Automation Conference",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1394--1395",

booktitle = "Proceedings of the 59th ACM/IEEE Design Automation Conference, DAC 2022",

}

TY - GEN

T1 - Late Breaking Results

T2 - 59th ACM/IEEE Design Automation Conference, DAC 2022

AU - Sun, Mengshu

AU - Li, Zhengang

AU - Lu, Alec

AU - Ma, Haoyu

AU - Yuan, Geng

AU - Xie, Yanyue

AU - Tang, Hao

AU - Li, Yanyu

AU - Leeser, Miriam

AU - Wang, Zhangyang

AU - Lin, Xue

AU - Fang, Zhenman

PY - 2022/7/10

Y1 - 2022/7/10

N2 - Vision transformers (ViTs) are emerging with significantly improved accuracy in computer vision tasks. However, their complex architecture and enormous computation/storage demand impose urgent needs for new hardware accelerator design methodology. This work proposes an FPGA-aware automatic ViT acceleration framework based on the proposed mixed-scheme quantization. To the best of our knowledge, this is the first FPGA-based ViT acceleration framework exploring model quantization. Compared with state-of-the-art ViT quantization work (algorithmic approach only without hardware acceleration), our quantization achieves 0.31% to 1.25% higher Top-1 accuracy under the same bit-width. Compared with the 32-bit floating-point baseline FPGA accelerator, our accelerator achieves around 5.6× improvement on the frame rate (i.e., 56.4 FPS vs. 10.0 FPS) with 0.83% accuracy drop for DeiT-base.

AB - Vision transformers (ViTs) are emerging with significantly improved accuracy in computer vision tasks. However, their complex architecture and enormous computation/storage demand impose urgent needs for new hardware accelerator design methodology. This work proposes an FPGA-aware automatic ViT acceleration framework based on the proposed mixed-scheme quantization. To the best of our knowledge, this is the first FPGA-based ViT acceleration framework exploring model quantization. Compared with state-of-the-art ViT quantization work (algorithmic approach only without hardware acceleration), our quantization achieves 0.31% to 1.25% higher Top-1 accuracy under the same bit-width. Compared with the 32-bit floating-point baseline FPGA accelerator, our accelerator achieves around 5.6× improvement on the frame rate (i.e., 56.4 FPS vs. 10.0 FPS) with 0.83% accuracy drop for DeiT-base.

UR - https://www.scopus.com/pages/publications/85137493403

UR - https://www.scopus.com/pages/publications/85137493403#tab=citedBy

U2 - 10.1145/3489517.3530618

DO - 10.1145/3489517.3530618

M3 - Conference contribution

AN - SCOPUS:85137493403

T3 - Proceedings - Design Automation Conference

SP - 1394

EP - 1395

BT - Proceedings of the 59th ACM/IEEE Design Automation Conference, DAC 2022

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 10 July 2022 through 14 July 2022

ER -

Late Breaking Results: FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization

Abstract

Publication series

Conference

Bibliographical note

Publisher link

Find It

Other files and links

Fingerprint

Cite this