Optimizing the Performance of Text Classification Models by Improving the Isotropy of the Embeddings using a Joint Loss Function

Joseph Attieh*, Abraham Zewoudie, Vladimir Vlassov, Adrian Flanagan, Tom Bäckström

*Tämän työn vastaava kirjoittaja

Tutkimustuotos: Artikkeli kirjassa/konferenssijulkaisussaConference article in proceedingsScientificvertaisarvioitu

2 Lataukset (Pure)

Abstrakti

Recent studies show that the spatial distribution of the sen-
tence representations generated from pre-trained language models is
highly anisotropic. This results in a degradation in the performance of
the models on the downstream task. Most methods improve the isotropy
of the sentence embeddings by refining the corresponding contextual
word representations, then deriving the sentence embeddings from these
refined representations. In this study, we propose to improve the quality
of the sentence embeddings extracted from the [CLS] token of the pre-
trained language models by improving the isotropy of the embeddings.
We add one feed-forward layer between the model and the downstream
task layers, and we train it using a novel joint loss function. The pro-
posed approach results in embeddings with better isotropy, that gener-
alize better on the downstream task. Experimental results on 3 GLUE
datasets with classification as the downstream task show that our pro-
posed method is on par with the state-of-the-art, as it achieves perfor-
mance gains of around 2–3% on the downstream tasks compared to the
baseline.
AlkuperäiskieliEnglanti
OtsikkoDocument Analysis and Recognition – ICDAR 2023 - 17th International Conference, Proceedings
ToimittajatGernot A. Fink, Rajiv Jain, Koichi Kise, Richard Zanibbi
KustantajaSpringer
Sivut121-136
Sivumäärä16
ISBN (elektroninen)978-3-031-41734-4
ISBN (painettu)978-3-031-41733-7
DOI - pysyväislinkit
TilaJulkaistu - 19 elok. 2023
OKM-julkaisutyyppiA4 Artikkeli konferenssijulkaisussa
TapahtumaInternational Conference on Document Analysis and Recognition - San Jose, Yhdysvallat
Kesto: 21 elok. 202326 elok. 2023
Konferenssinumero: 17

Julkaisusarja

NimiLecture notes in computer science
KustantajaSpringer
ISSN (painettu)0302-9743
ISSN (elektroninen)1611-3349

Conference

ConferenceInternational Conference on Document Analysis and Recognition
LyhennettäICDAR
Maa/AlueYhdysvallat
KaupunkiSan Jose
Ajanjakso21/08/202326/08/2023

Sormenjälki

Sukella tutkimusaiheisiin 'Optimizing the Performance of Text Classification Models by Improving the Isotropy of the Embeddings using a Joint Loss Function'. Ne muodostavat yhdessä ainutlaatuisen sormenjäljen.

Siteeraa tätä