A Multi-Role Unified Buffer Architecture for an On-Device Vision Transformer Accelerator

Abstract

On-device Vision Transformer (ViT) acceleration is often limited by on-chip memory and data movement. This work presents a multi-role unified buffer that dynamically reuses seven banks as input, weight, intermediate, score, and output storage. The on-chip $QK^{\mathrm {T}}$ –Softmax– $SV$ dataflow retains attention intermediates and reuses one processing-element (PE) array across stages. Building on prior integer formulations, the accelerator integrates divider-free register-transfer-level (RTL) LayerNorm and Softmax units using a compact Lite-LUT and fixed-point log-domain subtraction. In a controlled evaluation using an otherwise 32-bit floating-point (FP32) model, replacing division-based nonlinear units with divider-free realizations changes Top-1 accuracy from 70.84% to 70.73% for DeiT-Tiny and from 78.36% to 78.27% for DeiT-Small. The final configuration with 8-bit weights and activations (W8A8) achieves 70.25% and 77.16%, corresponding to drops of 1.78 and 2.67 percentage points from the FP32 references. Implemented on a Xilinx ZCU104 field-programmable gate array (FPGA), the accelerator operates at 300 MHz with 1024 multiply–accumulate (MAC) units. Board measurements under Linux with a batch size of one, including external dynamic random-access memory (DRAM) transfers, yield 155 frames per second (FPS) with 6.453 ms end-to-end latency for DeiT-Tiny and 46 FPS with 21.594 ms end-to-end latency for DeiT-Small, using 28 and 56 instances of 36-Kb block RAM (BRAM36), respectively. Relative to an analytical fixed-role allocation under the same bank granularity, the multi-role design reduces BRAM36 requirements by 56.3%, while buffer-control and bank-reconfiguration logic accounts for 4.3% of the lookup tables and 5.3% of the flip-flops in the full accelerator.