Across three benchmarks, FRAME outperforms strong baselines by up to 13% MRR in search and 7.6 percentage points in clustering accuracy, demonstrating FRAME's potential for enhancing automated UI design and testing tools.
Abstract
User Interface (UI) programming is challenging due to the complex abstraction gap between code and graphical software representations. To bridge this gap, UI programming tools often rely on screen retrieval and clustering, which require accurate similarity measures based on overlapping features. However, computing feature-oriented similarity is difficult because screens with similar functionality often exhibit design variations. To address this, we propose FRAME (ReinForced UseR InterfAce Screen EMbedding with Graphical Structural ComprEhension), a multi-modal, neuro-symbolic embedding technique. FRAME constructs symbolic, graph-based representations of UI components to encode salient relationships and capture feature patterns across different screens. It leverages large vision-language models for visual and lexical encoding, alongside a novel UI-specific computational geometry algorithm that enables weighted embedding propagation. Across three benchmarks, FRAME outperforms strong baselines by up to 13% MRR in search and 7.6 percentage points in clustering accuracy. A comprehensive ablation study further confirms the benefit of each component, demonstrating FRAME's potential for enhancing automated UI design and testing tools.
Multimodal Large Language Models (MLLMs) have demonstrated strong performance on the UI-to-code task, which aims to generate UI code from design mock-ups. However, when applied to long and complex websites, they often struggle with fragmented segmentation, redundant code generation for repetitive components, and frequent UI inconsistencies. To systematically investigate and address these challenges, we introduce ComUIBench, a new multi-page complex webpage benchmark with component annotations, designed to evaluate MLLMs' ability to generate reusable UI code in realistic website scenarios. Building upon this benchmark, we propose ComUICoder, a component-based UI code generation framework that emphasizes semantic-aware segmentation, code reuse, and fine-grained refinement. Specifically, ComUICoder incorporates (1) Hybrid Semantic-aware Block Segmentation for accurate UI semantic coherent block detection, (2) Visual-aware Graph-based Block Merge to consolidate structurally similar components within and across webpages for reusable implementation, and (3) Priority-based Element-wise Feedback to refine generated code and reduce element-level inconsistencies. Extensive experiments demonstrate that ComUICoder significantly improves overall generation quality and code reusability on complex multipage websites. Our datasets and code are publicly available at https://github.com/WebPAI/ComUICoder.
Jingyu Xiao, Jiantong Qin, ShuoQiu Li et al.· Proceedings of the 32nd ACM...· 1 citation
UI/UX design quality plays a critical role in user satisfaction, accessibility, and product success. However, detecting design anti-patterns such as inconsistent visual hierarchy, poor affordance, or misleading navigation remains largely manual, subjective, and error-prone. This paper proposes a multimodal machine learning framework that analyzes visual mockups, interaction flows, and textual design descriptions to automatically identify UI/UX anti-patterns. The system integrates computer vision models, layout-graph encoders, and text language models to detect structural, visual, and semantic inconsistencies across screens. A multimodal fusion model combines these signals to produce a consolidated anti-pattern risk score and label. We evaluate the approach using a curated dataset of annotated UI anti-patterns across mobile and web applications. Experimental results show that multimodal learning significantly outperforms unimodal baselines, especially for patterns requiring contextual or cross-screen reasoning. The findings highlight the potential of unified multimodal ML pipelines in automating UI/UX quality checks and supporting design governance.
Sithik Shah, Nickson prabhu G· International Journal of Inn...· 0 citations
AI-assisted developer tools increasingly mediate programming through chat panels, terminal agents, generated diffs, and streaming status output. These interaction surfaces may create visual accessibility barriers for blind, low-vision, and color-vision-deficient developers, yet little is known about how such barriers are reported in public tool ecosystems. We analyze issues and forum discussions from five AI developer tool ecosystems: GitHub Copilot in VS Code, Cursor, Claude Code, OpenAI Codex, and OpenCode. From 2,652 keyword-retrieved candidates, a three-model ensemble identified 600 unanimously positive visual accessibility reports. A stratified manual sanity check supported this conservative selection. Topic modeling and qualitative analysis identified three recurring categories: screen-reader and assistive-technology barriers; visual presentation, contrast, and differentiation problems; and readability, scaling, and control limitations in AI-specific interfaces. The relative prominence of these concerns varied across ecosystems and reflected differences in editor, terminal, chat, diff, and agent interaction surfaces. An exploratory metadata analysis further identified differences in reporter activity and, across the GitHub-based ecosystems, maintainer participation and closure processes. These findings show that the accessibility record of AI developer tools is shaped by both their interaction design and the reporting and maintenance practices of their surrounding ecosystems.
Existing screenshot-to-code systems face a trade-off between flexibility and controllability. Direct multimodal generation can hallucinate visible details, whereas structured pipelines reduce such errors through component-wise decomposition, predefined templates, and customized intermediate representations. These structures, however, introduce additional generative orchestration and restrict outputs to designs covered by the representation. We investigate whether selective tool grounding can improve the fidelity--efficiency trade-off of direct widget-to-code generation. We introduce \textbf{WidgetGen}, a lightweight tool-grounded framework that extracts observable text and color evidence, performs high-level layout and optional chart reasoning, and directly generates executable JavaScript XML (\emph{JSX}). This design reduces reliance on component-wise generation while avoiding a fixed UI schema. Across six multimodal models and \(1{,}000\) held-out widgets, WidgetGen outperforms direct prompting and the structured Widget2Code pipeline on most visual reconstruction metrics, with consistent gains in area, legibility, and style. Finally, reconstruction-derived image-code pairs improve six Qwen-family open-weight models across every reported metric through supervised fine-tuning. These results establish WidgetGen as a strong lightweight baseline and show that selective evidence grounding offers an effective alternative to extensive representation constraints.
Houston H. Zhang, Tao Zhang, Li Gu et al.· 0 citations
User-interface screenshots contain dense visual and semantic structure: text labels, icons, buttons, lists, cards, toolbars, form inputs, and navigation regions frequently coexist on one screen. This study presents a compact, evidence-fused pipeline for LLM-assisted UI element grounding and design-rationale cards. The pipeline proposes UI elements from screenshot-aligned hierarchy candidates, classifies their control types, grounds natural referring expressions to element boxes, infers layout-level design tags, and produces concise rationale cards whose language is constrained by structured evidence. Experiments used 1,460 mobile UI screenshots and corresponding JSON view hierarchies, yielding 31,968 parsed nodes and 30,101 labeled elements across 25 component categories. All tasks used screen-disjoint training, validation, and test partitions. The evidence-fused detector reached 0.950 AP50 and 0.998 Recall@100 on the test screens. The evidence-fused control classifier reached 0.932 accuracy and 0.942 weighted F1. Across 2,161 generated referring trials, multi-evidence grounding reached 0.876 Top-1 accuracy, 0.979 Top-3 accuracy, and 0.928 mean reciprocal rank. Predicted component labels substantially improved layout tagging over geometry-only rules for forms, navigation, cards, image-heavy screens, and advertisements. Across 643 cards per output condition, the evidence-card generator preserved role, position, and action evidence with a contradiction rate of 0.000 under deterministic consistency checks. This last result establishes internal agreement with the evidence record, not human-rated usefulness or trust. Overall, the findings show that screenshot geometry and hierarchy metadata can be combined into an inspectable grounding-and-explanation layer, while also indicating that hierarchy dependence, rare-class imbalance, generated grounding phrases, and the absence of user evaluation limit broader deployment claims.
Nathan Han· Journal of Science Innovatio...· 0 citations
Effective user interface (UI) design requires a harmonious balance between visual appeal, cognitive usability, and layout consistency. However, current automatic UI generation approaches primarily focus on visual appearance or component detection and lack a unified framework that integrates cognitive principles, color intelligence, and structural reasoning. Furthermore, existing methods suffer from limited layout generalization, poor interpretability, static color selection, and inconsistent behavior across screens. In this context, this work proposes an automated UI prototyping framework that integrates the Faster region-based convolutional neural network (Faster R-CNN)-based component detection and CIECAM02 uniform color space (CAM02-UCS)-driven perceptual color modeling, enriched with cognitive and visual design principles. The Faster R-CNN is used to identify UI components and infer hierarchical structures from large-scale interface datasets. An enhanced color generation module analyzes brand or reference images to ensure perceptually uniform, harmonious, and usability-compliant color themes using CAM02-UCS. These outputs are further optimized through cognitive design rules, including Gestalt grouping, Fitts' and Hick's laws, attention-based spacing, and visual hierarchy modeling, to automatically generate refined, task-oriented UI layouts. Experiments conducted on RICO, ENRICO, and Guo's UI Color Datasets show that the proposed system achieves 92.4% component detection accuracy, improves layout clarity and reading order accuracy by 18.7%, and produces color palettes rated 24.5% more harmonious by designers compared to baseline methods. User evaluations also indicate a 31% reduction in perceived cognitive load and a 28% increase in design consistency across screens. These findings demonstrate that combining deep learning-based structural understanding with perceptually grounded color modeling and cognitive design principles produces UI prototypes that are highly efficient, aesthetically coherent, and user-friendly. This framework establishes a novel, end-to-end approach to intelligent and human-centered automated UI prototyping.
Jun Shi, Di-Fei Chen· Journal of Visualized Experi...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
