Article Details

Abstract

Robust GUI element recognition is a prerequisite for building practical GUI agents. However, purely vision-based detectors such as YOLO-style models often struggle to balance precision and recall on real-world interfaces, while general-purpose large models such as ChatGPT offer strong semantic understanding but cannot reliably localize UI elements. To bridge this gap, this paper proposes a GUI element recognition enhancement framework that combines multi-source candidate generation with MLLM-based semantic verification. Specifically, it aggregates candidates from UIED and YOLOv8, then performs an initial fusion via consistency-driven clustering to form a high-recall proposal set. Next, it verifies each candidate using an MLLM and applies a dual-threshold scheme with geometric constraints to suppress background clutter and remove invalid proposals, yielding a refined set of UI elements. Experiments on the denoised RICO subset show that this method preserves a high recall (0.819) while improving overall performance, achieving a precision of 0.573 and an F1 score of 0.674. In terms of accuracy, it improves over UIED and YOLOv8 by approximately 20.7% and 13.8%, respectively. These results demonstrate that candidate-level semantic verification effectively complements conventional detectors and mitigates the localization limitations of general MLLMs, enabling more reliable downstream GUI agent execution.

Keywords

Multimodal Large Language Model; GUI element recognition; multisource candidate fusion; semantic verification

References

1. [1] WANG G Z，XIE Y Q，JIANG Y F，et al. Voyager: An Open-ended Embodied Agent with Large Language Models [J/OL].arXiv:2305.16291 [cs.AI].（2023-05-25）.https://arxiv.org/abs/2305.16291.
2. [2] WANG S，LIU W W，CHEN J X，et al. GUI Agents with Foundation Models: A Comprehensive Survey [J/OL].arXiv:2411.04890 [cs.AI].（2024-11-07）.https://arxiv.org/abs/2411.04890.
3. [3] JIANG Y. Computational Representations for Graphical User Interfaces [C]//CHI EA'24: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems.New York：ACM，2024：1-6.
4. [4] XIE M L，FENG S D，XING Z C，et al. UIED: A Hybrid Tool for GUI Element Detection [C]//Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering.New York：ACM，2020：1655-1659.
5. [5] SOHAN M，RAM T S，REDDY C V R. A Review on YOLOv8 and its Advancements [C]//International Conference on Data Intelligence and Cognitive Informatics.Singapore：Springer，2024：529-545.
6. [6] LIN T-Y，DOLLÁR P，GIRSHICK R，et al. Feature Pyramid Networks for Object Detection [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Honolulu：IEEE，2017：2117-2125.
7. [7] CHEN D P，CHEN R X，ZHANG S L，et al. MLLM-as-a-judge: Assessing Multimodal LLM-as-a-judge with Vision-language Benchmark [J/OL].arXiv:2402.04788 [cs.CL].（2024-02-07）.https://arxiv.org/abs/2402.04788.
8. [8] CHENG P Z，WU Z，WU Z R，et al. OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agents [J/OL].arXiv:2503.16465 [cs.HC].（2025-02-26）.https://arxiv.org/abs/2503.16465.
9. [9] WANG Z W，CHEN W Z，YANG L Y，et al. MP-GUI: Modality Perception with MLLMs for GUI Understanding [C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Nashville：IEEE，2025：29711-29721.
10. [10] DEKA B，HUANG Z F，FRANZEN C，et al. Rico: A Mobile APP Dataset for Building Data-driven Design Applications [C]//UIST'17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology.Québec City：ACM，2017：845-854.

About the authors

L ü Guizhou (2001-), male, Han ethnicity, from Chuzhou, Anhui Province, is currently pursuing a master's degree in GUI automation testing.