Article Details

Abstract

In complex environments, traditional facial expression recognition methods suffer from poor generalization and can hardly balance real-time performance with deep semantic understanding. To address these problems, this study designs and implements a hybrid recognition system based on YOLO11 and Qwen2.5-VL. The system adopts a modular hierarchical architecture: it first uses YOLO11 to detect face regions and extract multi-scale features, and then builds a dual-modal recognition mechanism that combines lightweight CNN classification with Qwen2.5-VL multimodal analysis. Experimental results show that the system achieves a Top-1 accuracy of 84.80% on the FER2013 dataset and maintains a frame rate above 10 FPS in real-time camera detection. While ensuring detection accuracy, the system overcomes the limitation of traditional classification models that output only discrete labels and can generate emotional semantic descriptions rich in contextual information, thus providing a more robust technical solution for affective human-computer interaction in complex environments.

Keywords

facial expression recognition; YOLO11; Qwen2.5-VL; multimodal analysis; real-time detection

References

1. [1] Khoi C M，Thien L D，Khang N P H，et al. Real-Time Driver Drowsiness Detection and Alert System Using YOLO11 on a Jetson Nano[C]//2025 19th International Conference on Advanced Computing and Analytics (ACOMPA)，2025：15-21.
2. [2] Li S，Deng W. Deep Facial Expression Recognition: A Survey[J] IEEE Transactions on Affective Computing，2022，13（3）：1195-1215.
3. [3] Cai J，Meng Z，Khan A S，et al. Probabilistic Attribute Tree Structured Convolutional Neural Networks for Facial Expression Recognition in the Wild[J].IEEE Transactions on Affective Computing，2022，14（3）：1927-1941.
4. [4] Aymon U，Kamarudin N S，Nasir A F A. Facial Expression Recognition with YOLOv11 and YOLOv12: A Comparative Study[C]//2025 IEEE 9th International Conference on Software Engineering and Computer Systems (ICSECS)，2025：18-23.
5. [5] Yu Z，Huang H，Chen W，et al. YOLO-FaceV2: A Scale and Occlusion Aware Face Detector[J/OL].Pattern Recognition，2024，155：110714.https://doi.org/10.1016/j.patcog.2024.110714.
6. [6] Bai S，Chen K，Liu X，et al. Qwen2.5-VL Technical Report[PP/OL].arXiv.[2026-06-01].https://doi.org/10.48550/arXiv.2502.13923.
7. [7] Lian Z，Chen H，Chen L，et al. AffectGPT: A New Dataset,Model,and Benchmark for Emotion Understanding with Multimodal Large Language Models[PP/OL].arXiv.[2026-06-01].https://doi.org/10.48550/arXiv.2501.16566.
8. [8] Wu J，Gan W，Chen Z，et al. Multimodal Large Language Models: A Survey[C]//2023 IEEE International Conference on Big Data (BigData)，2023：2247-2256.
9. [9] Goodfellow I J，Erhan D，Carrier P L，et al. Challenges in Representation Learning: A Report on Three Machine Learning Contests[J].Neural Networks，2015，64：59-63.
10. [10] Li Y，Wei J，Liu Y，et al. Deep Learning for Micro Expression Recognition: A Survey[J].IEEE Transactions on Affective Computing，2022，13（4）：2028-2046.

About the authors

Zhang Chen (1995-), female, Han ethnicity, from Heze, Shandong, teaching assistant, master's student, research interests include deep learning and computer vision.