Abstrakti
Creating open-domain chatbots requires large amounts of conversational data and related benchmark tasks to evaluate them. Standardized evaluation tasks are crucial for creating automatic evaluation metrics for model development; otherwise, comparing the models would require resource-expensive human evaluation. While chatbot challenges have recently managed to provide a plethora of such resources for English, resources in other languages are not yet available. In this work, we provide a starting point for Finnish open-domain chatbot research. We describe our collection efforts to create the Finnish chat conversation corpus FinChat, which is made available publicly. FinChat includes unscripted conversations on seven topics from people of different ages. Using this corpus, we also construct a retrieval-based evaluation task for Finnish chatbot development. We observe that off-the-shelf chatbot models trained on conversational corpora do not perform better than chance at choosing the right answer based on automatic metrics, while humans can do the same task almost perfectly. Similarly, in a human evaluation, responses to questions from the evaluation set generated by the chatbots are predominantly marked as incoherent. Thus, FinChat provides a challenging evaluation set, meant to encourage chatbot development in Finnish.
Alkuperäiskieli | Englanti |
---|---|
Otsikko | Proceedings of Interspeech |
Kustantaja | International Speech Communication Association (ISCA) |
Sivut | 429-433 |
Sivumäärä | 5 |
Vuosikerta | 2020-October |
DOI - pysyväislinkit | |
Tila | Julkaistu - 2020 |
OKM-julkaisutyyppi | A4 Artikkeli konferenssijulkaisussa |
Tapahtuma | Interspeech - Shanghai, Kiina Kesto: 25 lokak. 2020 → 29 lokak. 2020 Konferenssinumero: 21 http://www.interspeech2020.org/ |
Julkaisusarja
Nimi | Interspeech |
---|---|
ISSN (elektroninen) | 1990-9772 |
Conference
Conference | Interspeech |
---|---|
Lyhennettä | INTERSPEECH |
Maa/Alue | Kiina |
Kaupunki | Shanghai |
Ajanjakso | 25/10/2020 → 29/10/2020 |
www-osoite |
Sormenjälki
Sukella tutkimusaiheisiin 'FinChat: Corpus and evaluation setup for Finnish chat conversations on everyday topics'. Ne muodostavat yhdessä ainutlaatuisen sormenjäljen.Tietoaineistot
-
Finnish conversational chat corpus, source
Kurimo, M. (Creator), Language Bank of Finland, 2022
http://urn.fi/urn:nbn:fi:lb-2022060801
Tietoaineisto: Dataset