Optimizing the Convolution Operation to Accelerate Deep Neural Networks on FPGA

Yufei Ma, Yu Cao, Sarma Vrudhula, Jae Sun Seo

Research output: Contribution to journalArticlepeer-review

252 Scopus citations


As convolution contributes most operations in convolutional neural network (CNN), the convolution acceleration scheme significantly affects the efficiency and performance of a hardware CNN accelerator. Convolution involves multiply and accumulate operations with four levels of loops, which results in a large design space. Prior works either employ limited loop optimization techniques, e.g., loop unrolling, tiling, and interchange, or only tune some of the design variables after the accelerator architecture and dataflow are already fixed. Without fully studying the convolution loop optimization before the hardware design phase, the resulting accelerator can hardly exploit the data reuse and manage data movement efficiently. This paper overcomes these barriers by quantitatively analyzing and optimizing the design objectives (e.g., memory access) of the CNN accelerator based on multiple design variables. Then, we propose a specific dataflow of hardware CNN acceleration to minimize the data communication while maximizing the resource utilization to achieve high performance. The proposed CNN acceleration scheme and architecture are demonstrated by implementing end-to-end CNNs including NiN, VGG-16, and ResNet-50/ResNet-152 for inference. For VGG-16 CNN, the overall throughputs achieve 348 GOPS and 715 GOPS on Intel Stratix V and Arria 10 FPGAs, respectively.

Original languageEnglish (US)
Pages (from-to)1354-1367
Number of pages14
JournalIEEE Transactions on Very Large Scale Integration (VLSI) Systems
Issue number7
StatePublished - Jul 2018
Externally publishedYes

Bibliographical note

Publisher Copyright:
© 2018 IEEE.


  • Accelerator architectures
  • convolutional neural networks (CNNs)
  • field-programmable gate array (FPGA)
  • neural network hardware


Dive into the research topics of 'Optimizing the Convolution Operation to Accelerate Deep Neural Networks on FPGA'. Together they form a unique fingerprint.

Cite this