Keras LSTM fit underfitting

Question

I have time series training data of about 5000 numbers. For each 100 numbers, I am trying to predict the 101st. At the end of the series, I would put in the predicted numbers back into the model to predict ahead of the time series. The attached graph shows the training data, the test data and the prediction output. Currently, the model seems to be under-fitting. I would like to know what hyperparameters should be changed, or if I need to re-structure my input and output data.

I am using the following LSTM network.

model = Sequential()
model.add(LSTM(128, input_shape=([bl,1]), activation='relu', return_sequences=True))
model.add(Dropout(0.1))
model.add(LSTM(128, return_sequences=True))
model.add(Dropout(0.1))
model.add(Flatten())
model.add(Dense(20,activation='relu'))
model.add(Dense(1))

model.compile(optimizer=adam(lr=0.0001), loss='mean_squared_error', metrics=['accuracy'])

model.fit(y_ba_tr_in, y_ba_tr_out,
          epochs=20,
          batch_size=5,shuffle=False,verbose=2)

y_ba_tr_in.shape = (4961, 100, 1)

y_ba_tr_out.shape = (4961, 1)

Answer 1

Something you could try is taking return_sequences=True out of your last LSTM layer. I believe this is generally the approach when you intend to predict for the next timestep.

After that modification, you also shouldn't need the subsequent Flatten() and Dense() layers.