7. Simple RNN
This chapter describes simple RNNs.
While simple RNNs have limitations in learning long-term dependencies, they are valuable for educational purposes to understand more advanced RNN architectures like LSTMs and GRUs.
Chapter Contents
Note
In this document, we will use diagrams like Fig.7-1 to illustrate the operation of the RNN.
For example, Fig.7-1 shows that an input sequence {$x^{(0)},x^{(1)},\ldots,x^{(T)}$} is fed into the RNN sequentially, starting from the first element.
Fig.7-1: Many-to-One Simple-RNN with Dense Layer
It is important to note that although Fig.7-1 depicts several RNN units, a single RNN unit receives the input sequence sequentially. In other words, the same unit is used to process each element of the sequence one after another. See Fig.7-2 for further clarification.
