Deeplabv3 validation loss is nan

Question

I had around 360 images splitted %25 as validation data. I could train Deeplabv3 with those images without any issue. Later on I have added around 40 images with labeled images. But the model now started to give validation loss always nan. Sometimes it gives at very first epoch some validation loss value but starting by second epoch the validation loss is always nan. The strange thing is I can still train Unet or any other model with the same data, without having no problem. And Later I discarded those 40 images and trained Deeplabv3 and worked again without any issue. I have checked labels and everything from those images and looks like there is no problem with new images. Any idea about what could cause this issue ?