Difference between hidden dimension and n_layers in rnn using pytorch

Question

I am stuck between hidden dimension and n_layers. What I understood so far, is that n_layers in the parameters of RNN using pytorch, is number of hidden layers. If n_layers represents the number if hidden layers than what is hidden dimension?

Answer 1

Actually documentation is really clear about their differences. Hidden size is number of features of the hidden state for RNN. So if you increase hidden size then you compute bigger feature as hidden state output.

However, num_layers is just multiple RNN units which contain hidden states with given hidden size.

num_layers=2 would mean stacking two RNNs together to form a stacked RNN, with the second RNN taking in outputs of the first RNN and computing the final results

I want to explain how RNN works with the image below. Each RNN unit (blue rectangles) takes one h_n (hidden state) and one input. Hidden dimension determines the feature vector size of the h_n (hidden state). At each timestep (t, horizontal propagation in the image) your rnn will take a h_n and input. Then if you have n_layers >1 it will create a intermediate output and give it to the upper layer(vertical). So hidden dimension determine the size of horizontal h_n in the image, whereas num_layers determine the number of blue cells in vertical axis in the image.

Answer 2

As @Alperen Kantarc? answer stated the hidden dimension is the number of parameters/features that each h(hidden state) stores (hidden state -> red circle) and n_layers are the rows of blue units (yellow rectangles) w=n_layers.

You only get the output from the topmost layer.

The following link may be useful: https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html