Article Details

Abstract

In order to better inherit traditional culture shadow play, a multi-dimensional interactive intelligent shadow play system based on human pose estimation is proposed, including shadow play robot and virtual shadow play person. The system applies a human pose estimation network, uses a motion algorithm to calculate the rotation angle of the bone, and designs a human-like mechanical structure applied to the shadow play, which can interact with the audience in multiple dimensions. In order to optimize the fluency of the shadow play system, this paper proposes a three-dimensional key point optimization scheme and anti-occlusion strategy based on perspective projection constraint for the characteristics of shadow play performances. In the output, the Kalman filter is added, the model is lightweight on Jetson AGX, and TensorRT acceleration is realized. After processing by these modules, the recognition accuracy and action fluency of the shadow play system are improved by no less than 30%, and the frame rate reaches 33 FPS.

Keywords

Deep Learning; human pose estimation; multi-dimensional interaction; shadow play

References

1. [1] Ouyang Junxi, Cui Chunxue Cultivation of Traditional Chinese Culture and Socialist Core Values [J]. Shandong Social Sciences, 2013 (3): 11-15
2. [2] Li Zonggui On the Connotation of Excellent Traditional Chinese Culture [J]. Academic Research, 2013 (11): 35-39
3. [3] Zhang Yu, Wen Guangzhao, Mi Siya, etc A review of two-dimensional human pose estimation based on deep learning [J]. Journal of Software, 2022, 33 (11): 4173-4191
4. [4] Kong Yinghui, Qin Yinfeng, Zhang Ke A review of deep learning two-dimensional human pose estimation methods [J]. Chinese Journal of Image and Graphics, 2023, 28 (7): 1965-1989
5. [5] Cao Xiaoyu, Xia Duanfeng A review of human pose estimation methods based on deep learning [J]. Modern Information Technology, 2022, 6 (23): 1-6
6. [6] Liao Haiyan, Wang Jingyi, Xi Zhiyuan, etc A scalable robot device for shadow puppetry performance [J]. Electronic Production, 2022, 30 (18): 83-85+89
7. [7] Cui Xin, Wang Xinhua, Xu Yin, etc Intelligent Shadow Puppetry Performance System with Mechatronics Integration [J]. Electronic Products World, 2020, 27 (9): 50-52
8. [8] Ni Guangxing, Xu Hua, Wang Chao Research on gesture recognition by integrating YOLOv5 and Mediapipe [J]. Computer Engineering and Applications, 2024, 60 (7): 108-118
9. [9] Chen Jingyu, Xu Jin, Luo Rong, etc A 3D human-computer interaction system based on gesture recognition [J]. Modern Information Technology, 2023, 7 (22): 88-91
10. [10] Wang Feng Research and Design of sensorless brushless DC motor control system [D]. Suzhou: Soochow University, 2020
11. [11] LIU Z C，LI Y D，ZHENG Z D. A Review of Drive Techniques for Multiphase Machines [J].CES Transactions on Electrical Machines and Systems，2018，2（2）： 243-251.
12. [12] Wang Chu Research on Motion Planning and Optimization Algorithm of Humanoid Robot Arm [D]. Hangzhou: Zhejiang University, 2018
13. [13] Yuan Mengen Robot arm motion planning algorithm based on monocular vision estimation [D]. Kaifeng: Henan University, 2020
14. [14] Yellow algae bloom Research on Motion Control Algorithm of Robotic Arm under Multiple Constraints [D]. Hangzhou: Zhejiang University, 2016
15. [15] Zhu Yuhui Design of Robot Arm Human Computer Interaction Control System Based on Kinect [D]. Mianyang: Southwest University of Science and Technology, 2016
16. [16] Zhang Junning, Su Qunxing, Liu Pengyuan, etc A monocular 3D object detection network based on perspective projection [J]. Robotics, 2020, 42 (3): 278-288
17. [17] Fu Huitong Research on Lightweight Model for Object Recognition Based on Deep Learning [D]. Xi'an: Xi'an University of Technology, 2022
18. [18] Yuan Zheming, Yuan Hongjie, Yan Yuxuan, etc A lightweight field insect recognition and classification model based on deep learning [J]. Journal of Jilin University: Engineering Edition, 2021, 51 (3): 1131-1139
19. [19] XIONG B，SUN Z Z，WANG J，et al. A Lightweight Model for Ship Detection and Recognition in Complex-Scene SAR Images [J]. Remote Sensing，2022，14（23）：6053.
20. [20] LIU Z Q，DING D. TensorRT Acceleration Based on Deep Learning OFDM Channel Compensation [J/OL].Journal of Physics: Conference Series，2022，2303（1）：012047[2025-02-14].https://iopscience.iop.org/article/10.1088/1742-6596/2303/1/012047.
21. [21] YEGULALP S. Nvidia's New TensorRT Speeds Machine Learning Predictions [EB/OL].[2025-02-14].https://www.infoworld.com/article/2253746/nvidias-new-tensorrt-speeds-machine-learning-predictions.html.

Funding

Key areas of artificial intelligence in Guangdong ordinary universities (2019KZDZX1017); Guangdong Province Science and Technology Innovation Strategy Special Fund (Climbing Plan Special Fund) (pdjh2024a375); Jiangmen Basic and Theoretical Science Research Science and Technology Plan Project (Jiangke [2021] No. 87 (2021030101900004391)); Wuyi University Innovation and Entrepreneurship Fund (2023CY07)

About the authors

Zhao Jianwen (2004-), male, Han ethnicity, from Jiangmen, Guangdong Province, is currently pursuing a master's degree. His research interests include deep learning theory and applications, as well as computer vision; Zeng Junying (1977-), male, Han ethnicity, from Ganzhou, Jiangxi Province, professor, doctoral degree, research interests: image processing, deep learning theory and application, information processing, machine vision; Corresponding author: Qin Chuanbo (1982-), male, Han ethnicity, from Suzhou, Anhui Province, associate professor, PhD, research direction: medical image processing, biometric recognition.