Novel Convolutional Transformer for Autonomous Driving

Abstract

A crucial component of an autonomous driving vehicle (ADV) is the machine learning, which is able to drive towards a desired destination. Nowadays, there are different paradigms addressing the development of ADV. On the one hand, the modular pipelines, which divide the driving task into sub-tasks such as perception and maneuver planning and control. On the other hand, the end-to-end driving approaches that try to learn a direct mapping from input raw sensor data to vehicle control signals. The later are relatively less studied, but are gaining popularity since they are simple and save computational cost. In this project, we focus on the end-to-end autonomous driving. How should representations from complementary modality be integrated for end-to-end autonomous driving? Recent multi-modal methods have shown that complementing RGB images with depth and semantics has the potential to improve driving performance. However, those methods can't fuse the multi-modal input well and may negatively affect driving performance. Therefore, in this project, we propose the multi-modal transformer to integrate the sensor information. We also apply the auto-regressive method for the temporal reasoning to predict the waypoint. Finally, in order to address the semantic gap between the waypoint values and senor features, we will tokenize the waypoint values and equip the model with the ability to learn the representation of them. We experimentally validate the efficacy of our approach in urban settings involving complex scenarios using the CARLA urban driving simulator. Our code is available in the attached files.