Paying Attention to Descriptions Generated by Image Captioning Models

Hamed Rezazadegan Tavakoli, Rakshith Shetty, Ali Borji, Jorma Laaksonen

Tutkimustuotos: Artikkeli kirjassa/konferenssijulkaisussaConference contributionScientificvertaisarvioitu

29 Sitaatiot (Scopus)
143 Lataukset (Pure)

Abstrakti

To bridge the gap between humans and machines in image understanding and describing, we need further insight into how people describe a perceived scene. In this paper, we study the agreement between bottom-up saliency-based visual attention and object referrals in scene description constructs. We investigate the properties of human-written descriptions and machine-generated ones. We then propose a saliency-boosted image captioning model in order to investigate benefits from low-level cues in language models. We learn that (1) humans mention more salient objects earlier than less salient ones in their descriptions, (2) the better a captioning model performs, the better attention agreement it has with human descriptions, (3) the proposed saliencyboosted model, compared to its baseline form, does not improve significantly on the MS COCO database, indicating explicit bottom-up boosting does not help when the task is well learnt and tuned on a data, (4) a better generalization is, however, observed for the saliency-boosted model on unseen data.
AlkuperäiskieliEnglanti
OtsikkoProceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017
KustantajaIEEE
Sivut2506-2515
ISBN (elektroninen)978-1-5386-1032-9
DOI - pysyväislinkit
TilaJulkaistu - 2017
OKM-julkaisutyyppiA4 Artikkeli konferenssijulkaisuussa
TapahtumaIEEE International Conference on Computer Vision - Venice, Italia
Kesto: 22 lokakuuta 201729 lokakuuta 2017

Julkaisusarja

NimiIEEE International Conference on Computer Vision
KustantajaIEEE
ISSN (painettu)1550-5499
ISSN (elektroninen)2380-7504

Conference

ConferenceIEEE International Conference on Computer Vision
LyhennettäICCV
MaaItalia
KaupunkiVenice
Ajanjakso22/10/201729/10/2017

Sormenjälki Sukella tutkimusaiheisiin 'Paying Attention to Descriptions Generated by Image Captioning Models'. Ne muodostavat yhdessä ainutlaatuisen sormenjäljen.

Siteeraa tätä