Exploring Contextual Representation and Multi-modality for End-to-end Autonomous Driving

Research output: Contribution to journalArticleScientificpeer-review

1 Citation (Scopus)
14 Downloads (Pure)

Abstract

Learning contextual and spatial environmental representations enhances autonomous vehicle’s hazard anticipation and decision-making in complex scenarios. Recent perception systems enhance spatial understanding with sensor fusion but often lack global environmental context. Humans, when driving, naturally employ neural maps that integrate various factors such as historical data, situational subtleties, and behavioral predictions of other road users to form a rich contextual understanding of their surroundings. This neural
map-based comprehension is integral to making informed decisions on the road. In contrast, even with their significant advancements, autonomous systems have yet to fully harness this depth of human-like contextual understanding. Motivated by this, our work draws inspiration from human driving patterns and seeks to formalize the sensor fusion approach within an end-to-end autonomous driving framework. We introduce a framework that integrates three cameras (left, right, and center) to emulate the human field of view, coupled with top-down bird-eye-view semantic data to enhance contextual representation. The sensor data is fused and encoded using a self-attention mechanism, leading to an auto-regressive waypoint prediction module. We treat feature representation as a sequential problem, employing a vision transformer to distill the contextual interplay between sensor modalities. The efficacy of the proposed method is experimentally evaluated in both open and closed-loop settings. Our method achieves displacement error by 0.67 m in open-loop settings, surpassing current methods by 6.9% on the nuScenes dataset. In closed-loop evaluations on CARLA’s Town05 Long and Longest6 benchmarks, the proposed method enhances driving performance, route completion, and reduces infractions.
Original languageEnglish
Article number108767
Number of pages13
JournalEngineering Applications of Artificial Intelligence
Volume135
Early online date10 Jun 2024
DOIs
Publication statusPublished - Sept 2024
MoE publication typeA1 Journal article-refereed

Keywords

  • Vision transformer
  • Imitation learning
  • Attention
  • Vision-centric autonomous driving
  • Contextual representation

Fingerprint

Dive into the research topics of 'Exploring Contextual Representation and Multi-modality for End-to-end Autonomous Driving'. Together they form a unique fingerprint.

Cite this