How to construct deep recurrent neural networks

Razvan Pascanu; Caglar Gulcehre; Kyunghyun Cho; Yoshua Bengio

How to construct deep recurrent neural networks

Razvan Pascanu
, Caglar Gulcehre
, Kyunghyun Cho
, Yoshua Bengio

University of Montreal

Tutkimustuotos: Artikkeli kirjassa/konferenssijulkaisussa › Konferenssiesitys › Scientific › vertaisarvioitu

Abstrakti

In this paper, we explore different ways to extend a recurrent neural network (RNN) to a deep RNN. We start by arguing that the concept of depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, however, we find three points of an RNN which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. Based on this observation, we propose two novel architectures of a deep RNN which are orthogonal to an earlier attempt of stacking multiple recurrent layers to build a deep RNN (Schmidhuber, 1992; El Hihi and Bengio, 1996). We provide an alternative interpretation of these deep RNNs using a novel framework based on neural operators. The proposed deep RNNs are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNNs benefit from the depth and outperform the conventional, shallow RNNs.

Alkuperäiskieli	Englanti
Tila	Julkaistu - 1 tammik. 2014
OKM-julkaisutyyppi	Ei sovellu
Tapahtuma	International Conference on Learning Representations - Banff, Kanada Kesto: 14 huhtik. 2014 → 16 huhtik. 2014 Konferenssinumero: 2

Conference

Conference	International Conference on Learning Representations
Lyhennettä	ICLR
Maa/Alue	Kanada
Kaupunki	Banff
Ajanjakso	14/04/2014 → 16/04/2014

Rahoitus

We would like to thank the developers of Theano (Bergstra et al., 2010; Bastien et al., 2012). We also thank Justin Bayer for his insightful comments on the paper. We would like to thank NSERC, Compute Canada, and Calcul Québec for providing computational resources. Razvan Pascanu is supported by a DeepMind Fellowship. Kyunghyun Cho is supported by FICS (Finnish Doctoral Programme in Computational Sciences) and “the Academy of Finland (Finnish Centre of Excellence in Computational Inference Research COIN, 251170)”.

Muut tiedostot ja linkit

Link to publication in Scopus

Siteeraa tätä

@conference{1ef272890441420cb0a1f7ec7abf0b15,

title = "How to construct deep recurrent neural networks",

abstract = "In this paper, we explore different ways to extend a recurrent neural network (RNN) to a deep RNN. We start by arguing that the concept of depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, however, we find three points of an RNN which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. Based on this observation, we propose two novel architectures of a deep RNN which are orthogonal to an earlier attempt of stacking multiple recurrent layers to build a deep RNN (Schmidhuber, 1992; El Hihi and Bengio, 1996). We provide an alternative interpretation of these deep RNNs using a novel framework based on neural operators. The proposed deep RNNs are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNNs benefit from the depth and outperform the conventional, shallow RNNs.",

author = "Razvan Pascanu and Caglar Gulcehre and Kyunghyun Cho and Yoshua Bengio",

year = "2014",

month = jan,

day = "1",

language = "English",

note = "International Conference on Learning Representations, ICLR ; Conference date: 14-04-2014 Through 16-04-2014",

}

TY - CONF

T1 - How to construct deep recurrent neural networks

AU - Pascanu, Razvan

AU - Gulcehre, Caglar

AU - Cho, Kyunghyun

AU - Bengio, Yoshua

N1 - Conference code: 2

PY - 2014/1/1

Y1 - 2014/1/1

N2 - In this paper, we explore different ways to extend a recurrent neural network (RNN) to a deep RNN. We start by arguing that the concept of depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, however, we find three points of an RNN which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. Based on this observation, we propose two novel architectures of a deep RNN which are orthogonal to an earlier attempt of stacking multiple recurrent layers to build a deep RNN (Schmidhuber, 1992; El Hihi and Bengio, 1996). We provide an alternative interpretation of these deep RNNs using a novel framework based on neural operators. The proposed deep RNNs are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNNs benefit from the depth and outperform the conventional, shallow RNNs.

AB - In this paper, we explore different ways to extend a recurrent neural network (RNN) to a deep RNN. We start by arguing that the concept of depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, however, we find three points of an RNN which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. Based on this observation, we propose two novel architectures of a deep RNN which are orthogonal to an earlier attempt of stacking multiple recurrent layers to build a deep RNN (Schmidhuber, 1992; El Hihi and Bengio, 1996). We provide an alternative interpretation of these deep RNNs using a novel framework based on neural operators. The proposed deep RNNs are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNNs benefit from the depth and outperform the conventional, shallow RNNs.

UR - https://www.scopus.com/pages/publications/85070857075

M3 - Paper

T2 - International Conference on Learning Representations

Y2 - 14 April 2014 through 16 April 2014

ER -

How to construct deep recurrent neural networks

Abstrakti

Conference

Rahoitus

Muut tiedostot ja linkit

Sormenjälki

Siteeraa tätä