LLMs’ morphological analyses of complex FST-generated Finnish words

Anssi Moisio, Mathias Creutz, Mikko Kurimo

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientificpeer-review

Abstract

Rule-based language processing systems have been overshadowed by neural systems in terms of utility, but it remains unclear whether neural NLP systems, in practice, learn the grammar rules that humans use. This work aims to shed light on the issue by evaluating state-of-the-art LLMs in a task of morphological analysis of complex Finnish noun forms. We generate the forms using an FST tool, and they are unlikely to have occurred in the training sets of the LLMs, therefore requiring morphological generalisation capacity. We find that GPT-4-turbo has some difficulties in the task while GPT-3.5turbo struggles and smaller models Llama2-70B and Poro-34B fail nearly completely.

Original languageEnglish
Title of host publicationCMCL 2024 - 13th Edition of the Workshop on Cognitive Modeling and Computational Linguistics, Proceedings of the Workshop
EditorsTatsuki Kuribayashi, Giulia Rambelli, Ece Takmaz, Philipp Wicke, Yohei Oseki
PublisherAssociation for Computational Linguistics
Pages242-254
Number of pages13
ISBN (Electronic)979-8-89176-143-8
DOIs
Publication statusPublished - 2024
MoE publication typeA4 Conference publication
EventWorkshop on Cognitive Modeling and Computational Linguistics - Bangkok, Thailand
Duration: 15 Aug 202415 Aug 2024

Publication series

NameCMCL 2024 - 13th Edition of the Workshop on Cognitive Modeling and Computational Linguistics, Proceedings of the Workshop

Conference

ConferenceWorkshop on Cognitive Modeling and Computational Linguistics
Abbreviated titleCMCL
Country/TerritoryThailand
CityBangkok
Period15/08/202415/08/2024

Fingerprint

Dive into the research topics of 'LLMs’ morphological analyses of complex FST-generated Finnish words'. Together they form a unique fingerprint.

Cite this