LDA-based context dependent recurrent neural network language model using document-based topic distribution of words

Md Akmal Haidar*, Mikko Kurimo

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

3 Citations (Scopus)

Abstract

Adding context information into recurrent neural network language models (RNNLMs) have been investigated recently to improve the effectiveness of learning RNNLM. Conventionally, a fast approximate topic representation for a block of words was proposed by using corpus-based topic distribution of word incorporating latent Dirichlet allocation (LDA) model. It is then updated for each subsequent word using an exponential decay. However, words could represent different topics in different documents. In this paper, we form document-based distribution over topics for each word using LDA model and apply it in the computation of fast approximate exponentially decaying features. We have shown experimental results on a well known Penn Treebank corpus and found that our approach outperforms the conventional LDA-based context RNNLM approach. Moreover, we carried out speech recognition experiments on Wall Street Journal corpus and achieved word error rate (WER) improvements over the other approach.

Original languageEnglish
Title of host publication2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017 - Proceedings
PublisherIEEE
Pages5730-5734
Number of pages5
ISBN (Electronic)9781509041176
DOIs
Publication statusPublished - 16 Jun 2017
MoE publication typeA4 Article in a conference publication
EventIEEE International Conference on Acoustics, Speech, and Signal Processing - New Orleans, United States
Duration: 5 Mar 20179 Mar 2017

Publication series

NameProceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing
PublisherIEEE
ISSN (Electronic)2379-190X

Conference

ConferenceIEEE International Conference on Acoustics, Speech, and Signal Processing
Abbreviated titleICASSP
CountryUnited States
CityNew Orleans
Period05/03/201709/03/2017

Keywords

  • language modeling
  • latent Dirichlet allocation
  • Recurrent neural network
  • speech recognition

Fingerprint Dive into the research topics of 'LDA-based context dependent recurrent neural network language model using document-based topic distribution of words'. Together they form a unique fingerprint.

Cite this