Recurrent neural network

recurrent neural networksrecurrentSimple recurrent networkElman networksJordan Recurrent Neural Networkrecurrent networkrecurrent networksrecurrent neural netsRNN
A recurrent neural network (RNN) is a class of artificial neural networks where connections between nodes form a directed graph along a temporal sequence.wikipedia
205 Related Articles

Feedforward neural network

feedforwardfeedforward neural networksfeedforward networks
Unlike feedforward neural networks, RNNs can use their internal state (memory) to process sequences of inputs.
As such, it is different from recurrent neural networks.

Long short-term memory

LSTMLong Short-term Memory (LSTM)long short term memory
Such controlled states are referred to as gated state or gated memory, and are part of long short-term memory networks (LSTMs) and gated recurrent units.This is also called Feedback Neural Network. Long short-term memory (LSTM) networks were discovered by Hochreiter and Schmidhuber in 1997 and set accuracy records in multiple applications domains.
Long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in the field of deep learning.

Hopfield network

Hopfield netHopfield networksHopfield neural network
Hopfield networks - a special kind of RNN - were discovered by John Hopfield in 1982.
A Hopfield network is a form of recurrent artificial neural network popularized by John Hopfield in 1982, but described earlier by Little in 1974.

Gated recurrent unit

GRUGRUs
Such controlled states are referred to as gated state or gated memory, and are part of long short-term memory networks (LSTMs) and gated recurrent units.This is also called Feedback Neural Network.
Gated recurrent units (GRUs) are a gating mechanism in recurrent neural networks, introduced in 2014 by Kyunghyun Cho et al. The GRU is like a long short-term memory (LSTM) with forget gate but has fewer parameters than LSTM, as it lacks an output gate.

Speech recognition

voice recognitionautomatic speech recognitionvoice command
This makes them applicable to tasks such as unsegmented, connected handwriting recognition or speech recognition.
Today, however, many aspects of speech recognition have been taken over by a deep learning method called Long short-term memory (LSTM), a recurrent neural network published by Sepp Hochreiter & Jürgen Schmidhuber in 1997.

Connectionist temporal classification

connectionist temporal classification (CTC)
In 2009, a Connectionist Temporal Classification (CTC)-trained LSTM network was the first RNN to win pattern recognition contests when it won several competitions in connected handwriting recognition.
Connectionist temporal classification (CTC) is a type of neural network output and associated scoring function, for training recurrent neural networks (RNNs) such as LSTM networks to tackle sequence problems where the timing is variable.

Sepp Hochreiter

Hochreiter
Long short-term memory (LSTM) networks were discovered by Hochreiter and Schmidhuber in 1997 and set accuracy records in multiple applications domains. Various methods for doing so were developed in the 1980s and early 1990s by Werbos, Williams, Robinson, Schmidhuber, Hochreiter, Pearlmutter and others.
LSTM overcomes the problem of recurrent neural networks (RNNs) and deep networks to forget information over time or, equivalently, through layers (vanishing or exploding gradient).

Jürgen Schmidhuber

J. SchmidhuberSchmidhuberJurgen Schmidhuber
Long short-term memory (LSTM) networks were discovered by Hochreiter and Schmidhuber in 1997 and set accuracy records in multiple applications domains. Various methods for doing so were developed in the 1980s and early 1990s by Werbos, Williams, Robinson, Schmidhuber, Hochreiter, Pearlmutter and others.
With his students Sepp Hochreiter, Felix Gers, Fred Cummins, Alex Graves, and others, Schmidhuber published increasingly sophisticated versions of a type of recurrent neural network called the long short-term memory (LSTM).

Jeffrey Elman

ElmanJeff ElmanElman, Jeffrey L.
An Elman network is a three-layer network (arranged horizontally as x, y, and z in the illustration) with the addition of a set of "context units" (u in the illustration).
In 1990, he introduced the simple recurrent neural network (SRNN), also known as the 'Elman network', which is capable of processing sequentially ordered stimuli, and has since become widely used.

Michael I. Jordan

JordanMichael JordanJordan, Michael I.
Jordan networks are similar to Elman networks.
In the 1980s Jordan started developing recurrent neural networks as a cognitive model.

Liquid state machine

Liquid-state machineLiquid-state machines
A variant for spiking neurons is known as a liquid state machine.
The recurrent nature of the connections turns the time varying input into a spatio-temporal pattern of activations in the network nodes.

Deep learning

deep neural networkdeep neural networksdeep-learning
Long short-term memory (LSTM) is a deep learning system that avoids the vanishing gradient problem.
Deep learning architectures such as deep neural networks, deep belief networks, recurrent neural networks and convolutional neural networks have been applied to fields including computer vision, speech recognition, natural language processing, audio recognition, social network filtering, machine translation, bioinformatics, drug design, medical image analysis, material inspection and board game programs, where they have produced results comparable to and in some cases superior to human experts.

Handwriting recognition

handwritinggesture alphabetshandwriting recognizer
This makes them applicable to tasks such as unsegmented, connected handwriting recognition or speech recognition. In 2009, a Connectionist Temporal Classification (CTC)-trained LSTM network was the first RNN to win pattern recognition contests when it won several competitions in connected handwriting recognition.
Since 2009, the recurrent neural networks and deep feedforward neural networks developed in the research group of Jürgen Schmidhuber at the Swiss AI Lab IDSIA have won several international handwriting competitions.

Spiking neural network

spiking neuronspikingspiking neural networks
A variant for spiking neurons is known as a liquid state machine.
This avoids the additional complexity of a recurrent neural network (RNN).

Language model

language modelingstatistical language modelsNeural network language models
LSTM broke records for improved machine translation, Language Modeling and Multilingual Language Processing.
The neural net architecture might be feed-forward or recurrent, and while the former is simpler the latter is more common.

Artificial neural network

artificial neural networksneural networksneural network
A recurrent neural network (RNN) is a class of artificial neural networks where connections between nodes form a directed graph along a temporal sequence.
Alternatively, networks that allow connections between neurons in the same or previous layers are known as recurrent networks''.

Convolutional neural network

convolutional neural networksCNNconvolutional
LSTM combined with convolutional neural networks (CNNs) improved automatic image captioning.
Long short-term memory (LSTM) recurrent units are typically incorporated after the CNN to account for inter-frame or inter-clip dependencies.

Vanishing gradient problem

vanishing gradientproblemsresidual
Long short-term memory (LSTM) is a deep learning system that avoids the vanishing gradient problem. A generative model partially overcame the vanishing gradient problem of automatic differentiation or backpropagation in neural networks in 1992.
Hochreiter's diploma thesis of 1991 formally identified the reason for this failure in the "vanishing gradient problem", which not only affects many-layered feedforward networks, but also recurrent networks.

Tony Robinson (speech recognition)

Tony RobinsonDr. Tony RobinsonRobinson
Various methods for doing so were developed in the 1980s and early 1990s by Werbos, Williams, Robinson, Schmidhuber, Hochreiter, Pearlmutter and others.
Tony Robinson is a pioneer in the application of recurrent neural networks to speech recognition, being one of the first to discover the practical capabilities of deep neural networks and how they can be used to benefit speech recognition.

Paul Werbos

Werbos
Various methods for doing so were developed in the 1980s and early 1990s by Werbos, Williams, Robinson, Schmidhuber, Hochreiter, Pearlmutter and others.
He also was a pioneer of recurrent neural networks.

Backpropagation through time

The standard method is called "backpropagation through time" or BPTT, and is a generalization of back-propagation for feed-forward networks.
Backpropagation through time (BPTT) is a gradient-based technique for training certain types of recurrent neural networks.

Recursive neural network

recursive neural netsrecursive neural tensor network
A recursive neural network is created by applying the same set of weights recursively over a differentiable graph-like structure by traversing the structure in topological order.
The gradient is computed using backpropagation through structure (BPTS), a variant of backpropagation through time used for recurrent neural networks.

Differentiable neural computer

differentiableDifferentiable neural computers
The combined system is analogous to a Turing machine or Von Neumann architecture but is differentiable end-to-end, allowing it to be efficiently trained with gradient descent.
It performed better than a traditional recurrent neural network.

Ronald J. Williams

Williams
Various methods for doing so were developed in the 1980s and early 1990s by Werbos, Williams, Robinson, Schmidhuber, Hochreiter, Pearlmutter and others.
He also made fundamental contributions to the fields of recurrent neural networks and reinforcement learning.

Chainer