Post-Attention Modulator for Dense Video Captioning

Zixin Guo*, Tzu-Jui Julius Wang, Jorma Laaksonen

*Tämän työn vastaava kirjoittaja

Tutkimustuotos: Artikkeli kirjassa/konferenssijulkaisussaConference article in proceedingsScientificvertaisarvioitu

33 Lataukset (Pure)


Dense video captioning (VC) aims at generating a paragraph-long description for events in video segments. Borrowing from the success in language modeling, Transformer-based models for VC have been shown effective also in modeling cross-domain video-text representations with cross-attention (Xatt). Despite Xatt’s effectiveness, the queries and outputs of attention, which are from different domains, tend to be weakly related. In this paper, we argue that the weak relatedness, or domain discrepancy, could impede a model from learning meaningful cross-domain representations. Hence, we propose a simple yet effective Post-Attention Modulator (PAM) that post-processes Xatt’s outputs to narrow the discrepancy. Specifically, PAM modulates and enhances the average similarity over Xatt’s queries and outputs. The modulated similarities are then utilized as a weighting basis to interpolate PAM’s outputs. In our experiments, PAM was applied to two strong VC baselines, VTransformer and MART, with two different video features on the well-known VC benchmark datasets ActivityNet Captions and YouCookII. According to the results, the proposed PAM brings consistent improvements in, e.g., CIDEr-D at most to 14.5%, as well as other metrics, BLEU and METEOR, considered.
OtsikkoProceedings of the 26th International Conference on Pattern Recognition (ICPR)
ISBN (elektroninen)978-1-6654-9062-7
DOI - pysyväislinkit
TilaJulkaistu - 2022
OKM-julkaisutyyppiA4 Artikkeli konferenssijulkaisussa
TapahtumaInternational Conference on Pattern Recognition - Montreal, Kanada
Kesto: 21 elok. 202225 elok. 2022
Konferenssinumero: 26


NimiInternational Conference on Pattern Recognition
ISSN (painettu)1051-4651


ConferenceInternational Conference on Pattern Recognition


Sukella tutkimusaiheisiin 'Post-Attention Modulator for Dense Video Captioning'. Ne muodostavat yhdessä ainutlaatuisen sormenjäljen.

Siteeraa tätä