Exploring Temporal Dependencies in Multimodal Referring Expressions with Mixed Reality

Elena Sibirtseva*, Ali Ghadirzadeh, Iolanda Leite, Mårten Björkman, Danica Kragic

*Tämän työn vastaava kirjoittaja

Tutkimustuotos: Artikkeli kirjassa/konferenssijulkaisussaConference article in proceedingsScientificvertaisarvioitu

4 Sitaatiot (Scopus)


In collaborative tasks, people rely both on verbal and non-verbal cues simultaneously to communicate with each other. For human-robot interaction to run smoothly and naturally, a robot should be equipped with the ability to robustly disambiguate referring expressions. In this work, we propose a model that can disambiguate multimodal fetching requests using modalities such as head movements, hand gestures, and speech. We analysed the acquired data from mixed reality experiments and formulated a hypothesis that modelling temporal dependencies of events in these three modalities increases the model’s predictive power. We evaluated our model on a Bayesian framework to interpret referring expressions with and without exploiting the temporal prior.

OtsikkoVirtual, Augmented and Mixed Reality. Applications and Case Studies - 11th International Conference, VAMR 2019, Held as Part of the 21st HCI International Conference, HCII 2019, Proceedings
ToimittajatJessie Y.C. Chen, Gino Fragomeni
ISBN (painettu)9783030215644
DOI - pysyväislinkit
TilaJulkaistu - 8 kesäk. 2019
OKM-julkaisutyyppiA4 Artikkeli konferenssijulkaisussa
TapahtumaInternational Conference on Virtual, Augmented and Mixed Reality - Orlando, Yhdysvallat
Kesto: 26 heinäk. 201931 heinäk. 2019
Konferenssinumero: 11


NimiLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Vuosikerta11575 LNCS
ISSN (painettu)0302-9743
ISSN (elektroninen)1611-3349


ConferenceInternational Conference on Virtual, Augmented and Mixed Reality


Sukella tutkimusaiheisiin 'Exploring Temporal Dependencies in Multimodal Referring Expressions with Mixed Reality'. Ne muodostavat yhdessä ainutlaatuisen sormenjäljen.

Siteeraa tätä