Mask rcnn is giving mAP less than 1% even though the training loss is less than 0.01

Question

I am trying to train a mask rcnn model using the tensorflow object detection api.

I am using custom dataset which is grey scale CT scan images of Lung of patients with lung cancer. While training the model using legacy/train.py, the total loss quickly converges to 0.01 in around 10 epochs. But viewing the training on tensorboard shows the Loss/BoxclassifierLoss/mask_loss is always 0. Running legacy/eval.py yielded a maximum of 0.0001 map, with the same images used to train the model. And the predicted boxes with 0.01 accuracy is no where near to the actual object.

I tried with matterport mask rcnn and getting good results and even trained a faster rcnn model from which I got a .28 map with same dataset.

And also I tried some augementation techniques in the config file.

can someone please explain why this is happening?