Enriched Pedestrian Crossing Prediction Using Carla Synthetic Data

Pedestrian crossing prediction, which involves anticipating whether a pedestrian will cross the street or not, is a crucial function in autonomous driving systems. This is also a safety requirement for the interaction of highly automated vehicles and pedestrians. The endeavours in this research domain heavily rely on processing videos captured by the frontal cameras of autonomous vehicles using advanced computer vision techniques and deep learning methods. While recent studies focus on the model architecture for crossing prediction by utilising pre-trained visual feature extractors, they often encounter challenges stemming from inaccurate input features such as pedestrian body pose and/or scene semantic information. In this study, we aim to enhance pose estimation and semantic segmentation algorithms by using synthetic data augmentation (SDA) and domain randomisation (DR) techniques. SDA allows for automatic annotations through predefined agents and objects in a simulated urban environment. However, it creates a domain gap between synthetic and real-world data. To tackle this, we introduce a DR technique to generate synthetic data mimicking various weather and ambient illumination conditions. We evaluated two training strategies on six algorithms for both pose estimation and semantic segmentation algorithms, and ultimately, we target four deep learning architectures for crossing prediction, including convolutional, recurrent, graph, and transformer neural networks. The proposed technique improves the extraction of pedestrian body pose and categorical semantic information, which in turn enhances the state-of-the-art. This results in effective feature selection as the input for the PIP task, improving prediction accuracy by 3.2%, 4.2%, and 6.3% to reach 87.6%, 92.2%, and 73.6% against the JAAD, PIE, and FU-PIP datasets, respectively. The study indicates that using a simulated environment with structural randomised properties can enhance the resilience of the pedestrian crossing prediction to variations in the input data.