Skip to content
Open access

Hardware-Aware Computational Analysis of Vision Transformer Attention Mechanisms

2026 · IEEE Access · Vol 14, pp. 156592-156608 · 0 citations · 44 references

Abstract

Transformers have become a dominant paradigm in modern deep learning, achieving state-of-the-art performance across computer vision, natural language processing, and generative artificial intelligence. However, their high computational and memory demands remain a major limitation, particularly for real-time and resource-constrained applications. Inspired by prior studies on Convolutional Neural Networks (CNNs), where systematic analysis of convolutional kernel designs has significantly improved efficiency and deployment, a similar perspective is extended to self-attention mechanisms in Vision Transformers (ViTs). A comprehensive benchmarking study is conducted on three representative vision transformer architectures: the standard ViT-B/16 with global self-attention, a ViT-B/16 incorporating a Performer-inspired kernel-based linear self-attention mechanism, and Swin-T with shifted-window self-attention. The analysis includes parameter count, FLOPs, memory access, inference time, power consumption, and classification accuracy, with particular emphasis on runtime decomposition and hardware-aware evaluation across edge-oriented platforms. The results indicate that kernel-based linear self-attention significantly reduces computational complexity compared with global self-attention while maintaining comparable memory access patterns, leading to improved real-time efficiency. In contrast, although Swin-T reduces FLOPs through shifted-window self-attention, it introduces increased memory access due to frequent data reorganization operations, limiting its practical performance gains on memory-bound hardware. These findings establish that reducing FLOPs alone is insufficient for real-time optimization and demonstrate that arithmetic intensity and memory access provide complementary insights for designing efficient next-generation self-attention mechanisms.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.