Multimodal end-to-end learning for autonomous steering in adverse road and weather conditions

Jyri Maanpää; Josef Taher; Petri Manninen; Leo Pakola; Iaroslav Melekhov; Juha Hyyppä

doi:10.1109/ICPR48806.2021.9413109

Multimodal end-to-end learning for autonomous steering in adverse road and weather conditions

Jyri Maanpää^*
, Josef Taher
, Petri Manninen
, Leo Pakola
, Iaroslav Melekhov
, Juha Hyyppä

^*Tämän työn vastaava kirjoittaja

National Land Survey of Finland

Tutkimustuotos: Artikkeli kirjassa/konferenssijulkaisussa › Conference article in proceedings › Scientific › vertaisarvioitu

16 Sitaatiot (Scopus)

Abstrakti

Autonomous driving is challenging in adverse road and weather conditions in which there might not be lane lines, the road might be covered in snow and the visibility might be poor. We extend the previous work on end-to-end learning for autonomous steering to operate in these adverse real-life conditions with multimodal data. We collected 28 hours of driving data in several road and weather conditions and trained convolutional neural networks to predict the car steering wheel angle from front-facing color camera images and lidar range and reflectance data. We compared the CNN model performances based on the different modalities and our results show that the lidar modality improves the performances of different multimodal sensor-fusion models. We also performed on-road tests with different models and they support this observation.

Alkuperäiskieli	Englanti
Otsikko	Proceedings of ICPR 2020 - 25th International Conference on Pattern Recognition
Kustantaja	IEEE
Sivut	699-706
Sivumäärä	8
ISBN (elektroninen)	9781728188089
DOI - pysyväislinkit	https://doi.org/10.1109/ICPR48806.2021.9413109
Tila	Julkaistu - 2020
OKM-julkaisutyyppi	A4 Artikkeli konferenssijulkaisussa
Tapahtuma	International Conference on Pattern Recognition - Virtual, Online, Milan, Italia Kesto: 10 tammik. 2021 → 15 tammik. 2021 Konferenssinumero: 25

Julkaisusarja

Nimi	Proceedings - International Conference on Pattern Recognition
Kustantaja	IEEE
ISSN (painettu)	1051-4651

Conference

Conference	International Conference on Pattern Recognition
Lyhennettä	ICPR
Maa/Alue	Italia
Kaupunki	Milan
Ajanjakso	10/01/2021 → 15/01/2021

Rahoitus

Academy of Finland projects (decisions 319011 and 318437) are gratefully acknowledged. Authors' contribution is the following: Maanp?? designed and performed the experiments and wrote the manuscript. Maanp??, Taher, Manninen, and Pakola took equal shares in instrumenting the autonomous driving platform and developing software. Maanp??, Taher, and Pakola participated to data collection. Melekhov advised in the model development and provided feedback on the manuscript. Hyypp? supervised the project. In addition the authors would like to thank Heikki Hyyti, Antero Kukko and Harri Kaartinen from the Finnish Geospatial Research Institute FGI and Juho Kannala from Aalto University School of Science for assistance and advice during this work.

Pääsy asiakirjaan

10.1109/ICPR48806.2021.9413109

https://arxiv.org/abs/2010.14924

Muut tiedostot ja linkit

Linkki julkaisuun Scopuksessa

Siteeraa tätä

Maanpää, J., Taher, J., Manninen, P., Pakola, L., Melekhov, I., & Hyyppä, J. (2020). Multimodal end-to-end learning for autonomous steering in adverse road and weather conditions. teoksessa Proceedings of ICPR 2020 - 25th International Conference on Pattern Recognition (Sivut 699-706). Artikkeli 9413109 (Proceedings - International Conference on Pattern Recognition). IEEE. https://doi.org/10.1109/ICPR48806.2021.9413109

@inproceedings{ca28b9a578c94384b0459fd6219c4db2,

title = "Multimodal end-to-end learning for autonomous steering in adverse road and weather conditions",

abstract = "Autonomous driving is challenging in adverse road and weather conditions in which there might not be lane lines, the road might be covered in snow and the visibility might be poor. We extend the previous work on end-to-end learning for autonomous steering to operate in these adverse real-life conditions with multimodal data. We collected 28 hours of driving data in several road and weather conditions and trained convolutional neural networks to predict the car steering wheel angle from front-facing color camera images and lidar range and reflectance data. We compared the CNN model performances based on the different modalities and our results show that the lidar modality improves the performances of different multimodal sensor-fusion models. We also performed on-road tests with different models and they support this observation.",

author = "Jyri Maanp{\"a}{\"a} and Josef Taher and Petri Manninen and Leo Pakola and Iaroslav Melekhov and Juha Hyypp{\"a}",

note = "Publisher Copyright: {\textcopyright} 2020 IEEE; International Conference on Pattern Recognition, ICPR ; Conference date: 10-01-2021 Through 15-01-2021",

year = "2020",

doi = "10.1109/ICPR48806.2021.9413109",

language = "English",

series = "Proceedings - International Conference on Pattern Recognition",

publisher = "IEEE",

pages = "699--706",

booktitle = "Proceedings of ICPR 2020 - 25th International Conference on Pattern Recognition",

address = "United States",

}

Maanpää, J , Taher, J, Manninen, P, Pakola, L, Melekhov, I & Hyyppä, J 2020, Multimodal end-to-end learning for autonomous steering in adverse road and weather conditions. julkaisussa Proceedings of ICPR 2020 - 25th International Conference on Pattern Recognition., 9413109, Proceedings - International Conference on Pattern Recognition, IEEE, Sivut 699-706, International Conference on Pattern Recognition, Milan, Italia, 10/01/2021. https://doi.org/10.1109/ICPR48806.2021.9413109

Multimodal end-to-end learning for autonomous steering in adverse road and weather conditions. / Maanpää, Jyri ; Taher, Josef; Manninen, Petri et al.
Proceedings of ICPR 2020 - 25th International Conference on Pattern Recognition. IEEE, 2020. s. 699-706 9413109 (Proceedings - International Conference on Pattern Recognition).

Tutkimustuotos: Artikkeli kirjassa/konferenssijulkaisussa › Conference article in proceedings › Scientific › vertaisarvioitu

TY - GEN

T1 - Multimodal end-to-end learning for autonomous steering in adverse road and weather conditions

AU - Maanpää, Jyri

AU - Taher, Josef

AU - Manninen, Petri

AU - Pakola, Leo

AU - Melekhov, Iaroslav

AU - Hyyppä, Juha

N1 - Conference code: 25

PY - 2020

Y1 - 2020

N2 - Autonomous driving is challenging in adverse road and weather conditions in which there might not be lane lines, the road might be covered in snow and the visibility might be poor. We extend the previous work on end-to-end learning for autonomous steering to operate in these adverse real-life conditions with multimodal data. We collected 28 hours of driving data in several road and weather conditions and trained convolutional neural networks to predict the car steering wheel angle from front-facing color camera images and lidar range and reflectance data. We compared the CNN model performances based on the different modalities and our results show that the lidar modality improves the performances of different multimodal sensor-fusion models. We also performed on-road tests with different models and they support this observation.

AB - Autonomous driving is challenging in adverse road and weather conditions in which there might not be lane lines, the road might be covered in snow and the visibility might be poor. We extend the previous work on end-to-end learning for autonomous steering to operate in these adverse real-life conditions with multimodal data. We collected 28 hours of driving data in several road and weather conditions and trained convolutional neural networks to predict the car steering wheel angle from front-facing color camera images and lidar range and reflectance data. We compared the CNN model performances based on the different modalities and our results show that the lidar modality improves the performances of different multimodal sensor-fusion models. We also performed on-road tests with different models and they support this observation.

UR - https://www.scopus.com/pages/publications/85110428750

U2 - 10.1109/ICPR48806.2021.9413109

DO - 10.1109/ICPR48806.2021.9413109

M3 - Conference article in proceedings

AN - SCOPUS:85110428750

T3 - Proceedings - International Conference on Pattern Recognition

SP - 699

EP - 706

BT - Proceedings of ICPR 2020 - 25th International Conference on Pattern Recognition

PB - IEEE

T2 - International Conference on Pattern Recognition

Y2 - 10 January 2021 through 15 January 2021

ER -

Maanpää J , Taher J, Manninen P, Pakola L, Melekhov I, Hyyppä J. Multimodal end-to-end learning for autonomous steering in adverse road and weather conditions. julkaisussa Proceedings of ICPR 2020 - 25th International Conference on Pattern Recognition. IEEE. 2020. s. 699-706. 9413109. (Proceedings - International Conference on Pattern Recognition). doi: 10.1109/ICPR48806.2021.9413109

Multimodal end-to-end learning for autonomous steering in adverse road and weather conditions

Abstrakti

Julkaisusarja

Conference

Rahoitus

Pääsy asiakirjaan

Muut tiedostot ja linkit

Sormenjälki

Siteeraa tätä