How to handel Imbalance data while using LSTM

Question

I am doing a project on an online signature verification system using RNNs LSTM. In the project, I am facing a problem while using the signatures as LSTM training data. I was using the SVC 2004 dataset where there are 40 signatures of each user. 20 are genuine and 20 are forged. among these signatures, each one is not of equal length. Some have 130 responses, some have 150 responses (rows) (the number of the column is same but the number of rows is different in each signature data). Both are signatures of the same person and I have to use both of them as training data. But each row is crucial so I can not downsample the data. also upsampling the data can affect the time dependency. Then How can I adjust this imbalance in the signatures? If anyone helps me solve this problem I will be really grateful to him/her. Thank you.