Transfer learning using tensorflow object detection api

Question

I'm trying to train the model using pretrained faster_rcnn_inception_v2_coco. I'm using the following config file:

model {
  faster_rcnn {
    num_classes: 37
    image_resizer {
      keep_aspect_ratio_resizer {
        min_dimension: 1080
        max_dimension: 1365
      }
    }
    feature_extractor {
      type: "faster_rcnn_inception_v2"
      first_stage_features_stride: 8
    }
    first_stage_anchor_generator {
      grid_anchor_generator {
        height_stride: 16
        width_stride: 16
        scales: 0.25
        scales: 0.5
        scales: 1.0
        scales: 2.0
        aspect_ratios: 0.5
        aspect_ratios: 1.0
        aspect_ratios: 2.0
      }
    }
    first_stage_box_predictor_conv_hyperparams {
      op: CONV
      regularizer {
        l2_regularizer {
          weight: 0.0001
        }
      }
      initializer {
        truncated_normal_initializer {
          stddev: 0.00999999977648
        }
      }
    }
    first_stage_nms_score_threshold: 0.0
    first_stage_nms_iou_threshold: 0.699999988079
    first_stage_max_proposals: 31
    second_stage_batch_size: 30
    first_stage_localization_loss_weight: 2.0
    first_stage_objectness_loss_weight: 1.0
    initial_crop_size: 14
    maxpool_kernel_size: 2
    maxpool_stride: 2
    second_stage_box_predictor {
      mask_rcnn_box_predictor {
        fc_hyperparams {
          op: FC
          regularizer {
            l2_regularizer {
              weight: 0.0001
            }
          }
          initializer {
            variance_scaling_initializer {
              factor: 1.0
              uniform: true
              mode: FAN_AVG
            }
          }
        }
        use_dropout: true
        dropout_keep_probability: 0.20
      }
    }
    second_stage_post_processing {
      batch_non_max_suppression {
        score_threshold: 0.0
        iou_threshold: 0.600000023842
        max_detections_per_class: 100
        max_total_detections: 100
      }
      score_converter: SOFTMAX
    }
    second_stage_classification_loss{
      weighted_sigmoid_focal{
        gamma:2
        alpha:0.5
      }
    }
    second_stage_localization_loss_weight: 2.0
    second_stage_classification_loss_weight: 1.0
  }
}
train_config {
  batch_size: 1
  data_augmentation_options {
    random_jitter_boxes {
    }
  }
  optimizer {
    adam_optimizer {
      learning_rate {
        manual_step_learning_rate {
          initial_learning_rate: 4.99999987369e-05
          schedule {
            step: 160000
            learning_rate: 1e-05
          }
          schedule {
            step: 175000
            learning_rate: 1e-06
          }
        }
      }
    }
    use_moving_average: true
  }
  gradient_clipping_by_norm: 10.0
  fine_tune_checkpoint: "/home/deploy/tensorflow/models/research/object_detection/ved/model.ckpt"
  from_detection_checkpoint: true
  num_steps: 400000
}
train_input_reader {
  label_map_path: "/home/deploy/tensorflow/models/research/object_detection/ved/tomato.pbtxt"
  tf_record_input_reader {
    input_path: "/home/deploy/tensorflow/models/research/object_detection/ved/train.record"
  }
}
eval_config {
  num_visualizations: 4
  max_evals: 5
  num_examples: 4
  max_num_boxes_to_visualize : 100
  metrics_set: "coco_detection_metrics"
  eval_interval_secs: 600
}
eval_input_reader {
  label_map_path: "/home/deploy/tensorflow/models/research/object_detection/ved/tomato.pbtxt"
  shuffle: true
  num_epochs: 1
  num_readers: 1
  tf_record_input_reader {
    input_path: "/home/deploy/tensorflow/models/research/object_detection/ved/val.record"
  }
  sample_1_of_n_examples: 2
}

But I'm getting following error:

InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [148] rhs shape= [40]
         [[node save/Assign_728 (defined at /home/deploy/tensorflow/models/research/object_detection/model_lib.py:490)  = Assign[T=DT_FLOAT, _class=["loc:@SecondStageBoxPredictor/BoxEncodingPredictor/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](SecondStageBoxPredictor/BoxEncodingPredictor/biases, save/RestoreV2/_1457)]]
         [[{{node save/RestoreV2/_1768}} = _Send[T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_1773_save/RestoreV2", _device="/job:localhost/replica:0/task:0/device:CPU:0"](save/RestoreV2:884)]]

I don't know why it's happening. I've changed num_classes, first_stage_max_proposals and second_stage_batch_size.

Answer 1

Try correcting the checkpoint file path. The checkpoint file should be from the same model that is used for training. Usually, it comes with the pre trained models package downloaded from TensorFlow Model Zoo.

Try to fix in this line:

  fine_tune_checkpoint: "/home/deploy/tensorflow/models/research/object_detection/ved/model.ckpt"

Answer 2

Hope this helps others trying to do Transfer learning using tensorflow object detection api

I found this Transfer learning with TensorFlow Hub, this link is about classification changing the code for object detection should be a nice learning curve for how every tries it out.

It basically has 3 steps

This tutorial demonstrates how to:

1. Use models from TensorFlow Hub with tf.keras.
2. Use an image classification model from TensorFlow Hub.
3. Do simple transfer learning to fine-tune a model for your own image classes.

You can have a look at the Downloading the model section which has a classifier as a example

You can check out pre trained object detection model that the Tensorflow Hub currently support

Here are a few good once's

Model name	Speed (ms)	COCO mAP
CenterNet HourGlass104 512x512	70	41.9
EfficientDet D2 768x768	67	41.8
EfficientDet D3 896x896	95	45.4
SSD ResNet101 V1 FPN 640x640 (RetinaNet101)	57	35.6
CenterNet Resnet101 V1 FPN 512x512	34	34.2
CenterNet Resnet50 V1 FPN 512x512	27	31.2

Step 1 and Step 2 can be completed if you follow the previous section properly.

You can use this section simple_transfer_learning

You have to got through the entire Transfer learning with TensorFlow Hub to understand more