Towards safe reinforcement learning-based traffic control via safety-layer action correction

Autonomous vehicles (AVs) are expected to significantly transform the operation of modern traffic systems, where their coexistence with human-driven vehicles (HDVs) gives rise to mixed-autonomy traffic. Moreover, advances in machine learning techniques have greatly improved the control capabilities of autonomous vehicles. In particular, reinforcement learning (RL) has been widely studied for autonomous vehicle control in traffic systems to improve efficiency in mixed-autonomy settings. However, the intrinsic need for trial-and-error in standard RL exploration makes it extremely difficult to deploy in real-world traffic environments, as learned policies may select unsafe actions that could lead to severe consequences. Conventional methods commonly incorporate large penalty terms into the reward function when safety constraints are violated; nevertheless, such approaches cannot guarantee the complete avoidance of violations during training. Inspired by the safety shield mechanism, we propose a safe exploration strategy integrated with the Proximal Policy Optimisation (PPO) algorithm for the AV to prevent constraint violations during RL training. We further evaluate the effectiveness of the proposed method in the Simulation of Urban Mobility (SUMO) simulator and compare it with a reward-shaping baseline in terms of cumulative reward and constraint violations. The simulation results demonstrate that our method achieves zero constraint violations while maintaining competitive training performance.