Automated UI Component Detection and Visual Layout Quality Analysis from Webpage Screenshots Using Deep Learning
TL;DR
The findings indicate that automated visual UI quality assessment is a tractable objective and that structurally consistent components are reliably detected, while underperforming classes such as button and navigation are primarily limited by class definition noise.
Abstract
This paper addresses the lack of automated, image-based tools for evaluating the visual layout quality of web user interfaces as existing approaches rely on source code and Document Object Model analysis rather than reasoning about the rendered interface as perceived by the end user. The study employs a YOLOv11s object detection model combined with a three criteria visual layout analysis pipeline covering clutter assessment, geometric alignment consistency and WCAG-based color contrast evaluation to detect and classify UI components and produce quantitative quality scores directly from webpage screenshots. The proposed method is evaluated on 46,131 processed samples from the WebUI dataset using standard object detection metrics including precision, recall, F1 score, mAP@0.5 and mAP@0.5:0.95. The results demonstrate an overall test set mAP@0.5 of 0.6236 and a precision of 0.7204 across 14 UI component categories, with individual class performance ranging from 0.9239 for footer to 0.2810 for button. The findings indicate that automated visual UI quality assessment is a tractable objective and that structurally consistent components are reliably detected, while underperforming classes such as button and navigation are primarily limited by class definition noise. This work contributes to the field of UI evaluation by providing a scalable, visual assessment framework that addresses actionable quality issues, serving as a practical tool for frontend developers, UI/UX designers and automated testing pipelines.