Siirry päänavigointiin Siirry hakuun Siirry pääsisältöön

Evaluating Text Summarization Techniques and Factual Consistency with Language Models

  • Md Moinul Islam*
  • , Usman Muhammad
  • , Mourad Oussalah
  • *Tämän työn vastaava kirjoittaja
  • University of Oulu

Tutkimustuotos: Artikkeli kirjassa/konferenssijulkaisussa › Conference article in proceedings › Scientific › vertaisarvioitu

3 Sitaatiot (Scopus)
213 Lataukset (Pure)

Abstrakti

Standard evaluation of automated text summarization (ATS) methods relies on manually crafted golden summaries. With the advances in Large Language Models (LLMs), it is legitimate to question whether these models can now potentially complement or replace human-crafted summaries. This study examines the effectiveness of several language models (LMs) in specifically addressing the issue of preserving factual consistency. By conducting a thorough assessment of various conventional and state-of-the-art performance metrics, such as ROUGE, BLEU, BERTScore, FActScore, and LongDocFACTScore across diverse datasets, our findings highlight the important relationship between linguistic eloquence and factual accuracy. The findings suggest that whereas LLMs, such as GPT and LLaMA, demonstrate considerable competence in producing concise and contextually-aware summaries, there remain difficulties in ensuring factual accuracy, particularly in domain-specific situations. Moreover, this work enhances the existing knowledge on summarization dynamics and highlights the need of developing more reliable and tailored evaluation techniques that minimize the probability of factual errors in text generated by ATS. In particular, the findings advance the current domain by providing a rigorous assessment of the balance between linguistic fluency and factual correct- ness, highlighting the limitations of current ATS frameworks and metrics to enhance the factual reliability of LM-generated summaries.

AlkuperäiskieliEnglanti
OtsikkoProceedings - 2024 IEEE International Conference on Big Data, BigData 2024
ToimittajatWei Ding, Chang-Tien Lu, Fusheng Wang, Liping Di, Kesheng Wu, Jun Huan, Raghu Nambiar, Jundong Li, Filip Ilievski, Ricardo Baeza-Yates, Xiaohua Hu
KustantajaIEEE
Sivut116-122
Sivumäärä7
ISBN (elektroninen)979-8-3503-6248-0
ISBN (painettu)979-8-3503-6249-7
DOI - pysyväislinkit
TilaJulkaistu - 2024
OKM-julkaisutyyppiA4 Artikkeli konferenssijulkaisussa
TapahtumaIEEE International Conference on Big Data - Washington, Yhdysvallat
Kesto: 15 jouluk. 2024 → 18 jouluk. 2024
https://www3.cs.stonybrook.edu/~ieeebigdata2024/

Julkaisusarja

NimiIEEE International Conference on Big Data
ISSN (elektroninen)2573-2978

Conference

ConferenceIEEE International Conference on Big Data
LyhennettäBigData
Maa/AlueYhdysvallat
KaupunkiWashington
Ajanjakso15/12/2024 → 18/12/2024
www-osoite

Sormenjälki

Sukella tutkimusaiheisiin 'Evaluating Text Summarization Techniques and Factual Consistency with Language Models'. Ne muodostavat yhdessä ainutlaatuisen sormenjäljen.

Siteeraa tätä