Article Details

Title · Authors · Abstract · Keywords · DOI · References

Improved Dynamic Exploration Noise Algorithm Based on Constrained TD3

CHEN Chunfu1, MU Yu2, HAN Kaitao1

(1.College of Computer Science and Technology, Taiyuan Normal University, Jinzhong 030619, China; 2.Taiyuan Institute of Technology, Taiyuan 030008, China)

Section：Computer Technology · Vol. 2025, Issue 7 · Pages：103-108 · 2096-4706（2025）07-0103-06 · CLC number：TP181；TP301.6；TP242 · Document code：A · DOI：10.19850/j.cnki.2096-4706.2025.07.019

Abstract

Aiming at the problem that unconstrained exploration may cause damage to the mobile car, this study proposes a Reinforcement Learning method that combines adaptive noise exploration and Lagrange multiplier constraints, aiming to optimize the trajectory planning of the car reaching the target point. This method improves the exploration efficiency by dynamically adjusting the noise, uses the TD3 algorithm to deal with the continuous action space, and uses the Lagrange multiplier method to deal with the constraints, which is different from the way of adding the penalty of unexpected behavior directly in the Markov Decision Process (MDP). Simulation experiments show that this method can effectively guide the car to avoid obstacles, reduce the violation of constraints, and ensure the safety and reliability of the task, showing good training convergence characteristics.

Keywords

Safety Reinforcement Learning; Constrained Markov Decision Process; trajectory planning; TD3 algorithm

References

  1. [1] Timing beeps, Huang Xihuan Overview of the Development and Application of Industrial Robot Technology [J]. Mechanical and Electrical Engineering, 2015, 32 (1): 1-13
  2. [2] Wang Xuesong, Wang Rongrong, Cheng Yuhu A Review of Security Reinforcement Learning [J]. Journal of Automation, 2023, 49 (9): 1813-1835
  3. [3] MORALES E F，ZARAGOZA J H. An Introduction to Reinforcement Learning [M]//SUCAR L E，MORALES E F，HOEY J. Decision Theory Models for Applications in Artificial Intelligence：Concepts and Solutions [S.I.]：IGI Global，2012：63-80.
  4. [4] ACHIAM J，HELD D，TAMAR A，et al. Constrained Policy Optimization [C]//Proceedings of the 34th International Conference on Machine Learning.Sydney：PMLR，2017，70：22-31.
  5. [5] SUI Y，GOTOVOS A，BURDICK J，et al. Safe Exploration for Optimization with Gaussian Processes [C]//International Conference on Machine Learning.Lille：PMLR，2015：997-1005.
  6. [6] LILLICRAP T P，HUNT J J，PRITZEL A，et al. Continuous Control with Deep Reinforcement Learning [J/OL].[2024-10-03].arXiv:1509.02971 [cs.LG].https://doi.org/10.48550/arXiv.1509.02971.
  7. [7] STOOKE A，ACHIAM J，ABBEEL P. Responsive Safety in Reinforcement Learning by PID Lagrangian Methods [J/OL].arXiv:2007.03964 [math.OC].[2024-10-06].https://doi.org/10.48550/arXiv.2007.03964.
  8. [8] IBARZ J，TAN J，FINN C，et al. How to Train your Robot with Deep Reinforcement Learning: Lessons we have Learned [J].The International Journal of Robotics Research，2021，40（4-5）：698-721.
  9. [9] MARCHESINI E，CORSI D，FARINELLI A. Benchmarking Safe Deep Reinforcement Learning in Aquatic Navigation [C]//2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).Prague：IEEE，2021：5590-5595.
  10. [10] Li Xiaoyu, Zhang Junhua, Guo Xiaoguang, etc Reinforcement Learning based Optimization Algorithm for Robot Chassis Energy Management and Path Planning [J]. Journal of Agricultural Engineering, 2024, 40 (21): 175-183
  11. [11] Nie Zhenbang, Yu Haibin Collision avoidance decision-making and trajectory tracking method for mobile robots based on reference trajectories [J]. Computer Integrated Manufacturing Systems, 2023, 29 (9): 2879-2889
  12. [12] ISCEN A，YU G，ESCONTRELA A，et al. Learning Agile Locomotion Skills with a Mentor [C]//2021 IEEE International Conference on Robotics and Automation (ICRA).Xi'an：IEEE，2021：2019-2025.
  13. [13] DALAL G，DVIJOTHAM K，VECERIK M，et al. Safe Exploration in Continuous Action Spaces [J/OL].arXiv:1801.08757 [cs.AI].[2024-10-10].https://doi.org/10.48550/arXiv.1801.08757.
  14. [14] RAMEZANI M，HABIBI H，SANCHEZ-LOPEZ J L，et al. UAV Path Planning Employing MPC-Reinforcement Learning Method Considering Collision Avoidance [C]//2023 International Conference on Unmanned Aircraft Systems (ICUAS).Warsaw：IEEE，2023：507-514.
  15. [15] JI J，ZHOU J，ZHANG B，et al. Omnisafe: An Infrastructure for Accelerating Safe Reinforcement Learning Research [J].Journal of Machine Learning Research，2024，25（285）：1-6.
  16. [16] ZHAO W Y，HE T N，CHEN R，et al. State-wise Safe Reinforcement Learning: A Survey [J/OL].arXiv:2302.03122 [cs.LG].[2024-0-16].https://doi.org/10.48550/arXiv.2302.03122.
  17. [17] FUJIMOTO S，HOOF H，MEGER D. Addressing Function Approximation Error in Actor-Critic Methods [J/OL].arXiv:1802.09477 [cs.AI].[2024-10-11].https://doi.org/10.48550/arXiv.1802.09477.

About the authors

Chen Chunfu (1999-), male, Han ethnicity, from Jinzhong, Shanxi Province, is currently pursuing a master's degree in reinforcement learning and robotics systems; Corresponding author: Mu Yu (1978-), male, Han ethnicity, from Taiyuan, Shanxi, lecturer, PhD, research direction: robot motion control and planning; Han Kaitao (1999-), male, Han ethnicity, from Lvliang, Shanxi Province, is currently pursuing a master's degree in optoelectronic information processing and communication technology.