TY - GEN
T1 - LLMs’ morphological analyses of complex FST-generated Finnish words
AU - Moisio, Anssi
AU - Creutz, Mathias
AU - Kurimo, Mikko
N1 - Publisher Copyright:
©2024 Association for Computational Linguistics.
PY - 2024
Y1 - 2024
N2 - Rule-based language processing systems have been overshadowed by neural systems in terms of utility, but it remains unclear whether neural NLP systems, in practice, learn the grammar rules that humans use. This work aims to shed light on the issue by evaluating state-of-the-art LLMs in a task of morphological analysis of complex Finnish noun forms. We generate the forms using an FST tool, and they are unlikely to have occurred in the training sets of the LLMs, therefore requiring morphological generalisation capacity. We find that GPT-4-turbo has some difficulties in the task while GPT-3.5turbo struggles and smaller models Llama2-70B and Poro-34B fail nearly completely.
AB - Rule-based language processing systems have been overshadowed by neural systems in terms of utility, but it remains unclear whether neural NLP systems, in practice, learn the grammar rules that humans use. This work aims to shed light on the issue by evaluating state-of-the-art LLMs in a task of morphological analysis of complex Finnish noun forms. We generate the forms using an FST tool, and they are unlikely to have occurred in the training sets of the LLMs, therefore requiring morphological generalisation capacity. We find that GPT-4-turbo has some difficulties in the task while GPT-3.5turbo struggles and smaller models Llama2-70B and Poro-34B fail nearly completely.
UR - http://www.scopus.com/inward/record.url?scp=85204300266&partnerID=8YFLogxK
U2 - 10.48550/arXiv.2407.08269
DO - 10.48550/arXiv.2407.08269
M3 - Conference article in proceedings
AN - SCOPUS:85204300266
T3 - CMCL 2024 - 13th Edition of the Workshop on Cognitive Modeling and Computational Linguistics, Proceedings of the Workshop
SP - 242
EP - 254
BT - CMCL 2024 - 13th Edition of the Workshop on Cognitive Modeling and Computational Linguistics, Proceedings of the Workshop
A2 - Kuribayashi, Tatsuki
A2 - Rambelli, Giulia
A2 - Takmaz, Ece
A2 - Wicke, Philipp
A2 - Oseki, Yohei
PB - Association for Computational Linguistics
T2 - Workshop on Cognitive Modeling and Computational Linguistics
Y2 - 15 August 2024 through 15 August 2024
ER -