Multimodal ML Models for Detecting Anti-Patterns in UI/UX Designs

Abstract

UI/UX design quality plays a critical role in user satisfaction, accessibility, and product success. However, detecting design anti-patterns such as inconsistent visual hierarchy, poor affordance, or misleading navigation remains largely manual, subjective, and error-prone. This paper proposes a multimodal machine learning framework that analyzes visual mockups, interaction flows, and textual design descriptions to automatically identify UI/UX anti-patterns. The system integrates computer vision models, layout-graph encoders, and text language models to detect structural, visual, and semantic inconsistencies across screens. A multimodal fusion model combines these signals to produce a consolidated anti-pattern risk score and label. We evaluate the approach using a curated dataset of annotated UI anti-patterns across mobile and web applications. Experimental results show that multimodal learning significantly outperforms unimodal baselines, especially for patterns requiring contextual or cross-screen reasoning. The findings highlight the potential of unified multimodal ML pipelines in automating UI/UX quality checks and supporting design governance.