A lightweight transformer-based real-time object detection method for edge devices

Abstract

Achieving high-accuracy, low-latency real-time object detection on resource-constrained edge devices is a core challenge in computer vision. Existing Transformer-based detectors are accurate, but their large parameter counts and computational cost hinder direct edge deployment. This paper proposes LT-Det, a lightweight Transformer-based real-time detector for edge devices, with three contributions: (1) a Lightweight Multi-Head Attention module (LMHA) that linearly approximates self-attention, reducing its complexity from O(n2) to O(n); (2) an Adaptive Feature Pyramid Network (AFPN) introducing deformable convolution into multi-scale fusion to improve small-object perception; and (3) an efficient Depth-wise Separable Convolution (DSConv) detection head that reduces inference latency. On COCO 2017 and VisDrone2023, LTDet attains 76.3% mAP@0.5 at 68.9 FPS on the NVIDIA Jetson Nano with only 2.600M parameters, outperforming comparable methods in both speed and accuracy.