How to construct deep recurrent neural networks

Razvan Pascanu; Caglar Gulcehre; Kyunghyun Cho; Yoshua Bengio

How to construct deep recurrent neural networks

Razvan Pascanu
, Caglar Gulcehre
, Kyunghyun Cho
, Yoshua Bengio

University of Montreal

Research output: Contribution to conference › Paper › Scientific › peer-review

Abstract

In this paper, we explore different ways to extend a recurrent neural network (RNN) to a deep RNN. We start by arguing that the concept of depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, however, we find three points of an RNN which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. Based on this observation, we propose two novel architectures of a deep RNN which are orthogonal to an earlier attempt of stacking multiple recurrent layers to build a deep RNN (Schmidhuber, 1992; El Hihi and Bengio, 1996). We provide an alternative interpretation of these deep RNNs using a novel framework based on neural operators. The proposed deep RNNs are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNNs benefit from the depth and outperform the conventional, shallow RNNs.

Original language	English
Publication status	Published - 1 Jan 2014
MoE publication type	Not Eligible
Event	International Conference on Learning Representations - Banff, Canada Duration: 14 Apr 2014 → 16 Apr 2014 Conference number: 2

Conference

Conference	International Conference on Learning Representations
Abbreviated title	ICLR
Country/Territory	Canada
City	Banff
Period	14/04/2014 → 16/04/2014

Funding

We would like to thank the developers of Theano (Bergstra et al., 2010; Bastien et al., 2012). We also thank Justin Bayer for his insightful comments on the paper. We would like to thank NSERC, Compute Canada, and Calcul Québec for providing computational resources. Razvan Pascanu is supported by a DeepMind Fellowship. Kyunghyun Cho is supported by FICS (Finnish Doctoral Programme in Computational Sciences) and “the Academy of Finland (Finnish Centre of Excellence in Computational Inference Research COIN, 251170)”.

Cite this

@conference{1ef272890441420cb0a1f7ec7abf0b15,

title = "How to construct deep recurrent neural networks",

abstract = "In this paper, we explore different ways to extend a recurrent neural network (RNN) to a deep RNN. We start by arguing that the concept of depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, however, we find three points of an RNN which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. Based on this observation, we propose two novel architectures of a deep RNN which are orthogonal to an earlier attempt of stacking multiple recurrent layers to build a deep RNN (Schmidhuber, 1992; El Hihi and Bengio, 1996). We provide an alternative interpretation of these deep RNNs using a novel framework based on neural operators. The proposed deep RNNs are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNNs benefit from the depth and outperform the conventional, shallow RNNs.",

author = "Razvan Pascanu and Caglar Gulcehre and Kyunghyun Cho and Yoshua Bengio",

year = "2014",

month = jan,

day = "1",

language = "English",

note = "International Conference on Learning Representations, ICLR ; Conference date: 14-04-2014 Through 16-04-2014",

}

TY - CONF

T1 - How to construct deep recurrent neural networks

AU - Pascanu, Razvan

AU - Gulcehre, Caglar

AU - Cho, Kyunghyun

AU - Bengio, Yoshua

N1 - Conference code: 2

PY - 2014/1/1

Y1 - 2014/1/1

N2 - In this paper, we explore different ways to extend a recurrent neural network (RNN) to a deep RNN. We start by arguing that the concept of depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, however, we find three points of an RNN which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. Based on this observation, we propose two novel architectures of a deep RNN which are orthogonal to an earlier attempt of stacking multiple recurrent layers to build a deep RNN (Schmidhuber, 1992; El Hihi and Bengio, 1996). We provide an alternative interpretation of these deep RNNs using a novel framework based on neural operators. The proposed deep RNNs are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNNs benefit from the depth and outperform the conventional, shallow RNNs.

AB - In this paper, we explore different ways to extend a recurrent neural network (RNN) to a deep RNN. We start by arguing that the concept of depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, however, we find three points of an RNN which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. Based on this observation, we propose two novel architectures of a deep RNN which are orthogonal to an earlier attempt of stacking multiple recurrent layers to build a deep RNN (Schmidhuber, 1992; El Hihi and Bengio, 1996). We provide an alternative interpretation of these deep RNNs using a novel framework based on neural operators. The proposed deep RNNs are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNNs benefit from the depth and outperform the conventional, shallow RNNs.

UR - https://www.scopus.com/pages/publications/85070857075

M3 - Paper

T2 - International Conference on Learning Representations

Y2 - 14 April 2014 through 16 April 2014

ER -

How to construct deep recurrent neural networks

Abstract

Conference

Funding

Other files and links

Fingerprint

Cite this