Compilation deep learning model is important if we unfreez the layers for fientuneing?

Question

I am classifying a medical images dataset into normal vs abnormal where I am applying transfer learning with ResNet50v2. I did a little change in the last layer and then for finetuning I unfreeze the layer. Search this type of query but couldn't find any of one. I am using Keras with TensorFlow. My question is:

1. Is it important to compile the model again after unfreezing layers?
2. If I save the checkpoints and load the best checkpoint, then after I save the model by using model.save() is this method okay for any future training.

# Create a MirroredStrategy.
import tensorflow as tf
from tensorflow import keras

def f1_m(y_true, y_pred):
    precision = precision_m(y_true, y_pred)
    recall = recall_m(y_true, y_pred)
    return 2*((precision*recall)/(precision+recall+K.epsilon()))


def recall_m(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    recall = true_positives / (possible_positives + K.epsilon())
    return recall

def precision_m(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    return precision
dependencies = {
    'f1_m': f1_m,
    'recall_m': recall_m,
    'precision_m': precision_m,

}
# I am using multi GPU and custom metrices so loading need these custom arguments.
strategy = tf.distribute.MirroredStrategy()
print("Number of devices: {}".format(strategy.num_replicas_in_sync))
# Open a strategy scope.
with strategy.scope():
    adam = tf.keras.optimizers.Adam(learning_rate=0.00005, amsgrad=True, name='Adam',)
    model = keras.models.load_model('/home/classifier/model/resnet50_-36.hdf5', custom_objects=dependencies)
    model.summary()

model.trainable = True
model.compile(loss='binary_crossentropy',optimizer=adam,metrics=metric)

Now here I compile the model and I am confused here that is it important to compile after unfreezing or do I have to save the checkpoint and then load the model and then unfreeze and start training?

Saving Model:

model.save("path")
# Load the save model like before
model.trainable = True
model.compile(loss='binary_crossentropy',optimizer=adam,metrics=metric)

Answer 1

Thank you @Noltibus for your answer but at that time I was looking for some technical details which I explored now and I am sharing my experiences here.

First thing first, we have two types of training while we use pre-trained models trained on imagenet dataset.

> Classifier Training:

While we are trying to change the final layer and train our initial model having no change in feature extraction layers. Then we select the best model during training and then finetune by unfreezing the convolutional layers.

> Fintune by Unfreezing the Convolutional Layers:

The best model is selected and then unfreeze the layers as much as you want and again compile the model and fit the model.

Code to train Classifier:

image_size = 512
input_shape = (image_size, image_size, 3)
pre_trained_model = tf.keras.applications.ResNet50V2(input_shape=input_shape, include_top=False, weights="imagenet")
for layer in pre_trained_model.layers:
    layer.trainable = False

gap = keras.layers.GlobalAveragePooling2D()(pre_trained_model.output, training=False)
output = keras.layers.Dense(1, activation='sigmoid')(gap)
model = keras.Model(inputs=pre_trained_model.input, outputs=output)
model.compile(loss='binary_crossentropy',
              optimizer=adam,
              metrics=metric)#'accuracy'

history = model.fit(train_gen,
                            use_multiprocessing=True,
                            workers=16,
                            epochs=50,     
                            class_weight=class_weights,
                            steps_per_epoch=train_steps,                    
                            validation_data=val_gen,
                            validation_steps=val_steps,                    
                            shuffle=True,
                            callbacks=call_backs)

The above code state that we just load a pre-trained model and then change its final layers and compile and fit the model.

Now we are doing fine-tuning on the same data to learn more features so when you unfreeze you have to again compile the model and then retrain. Keep in mind that you have to verify which layer you want to train it's up to your problem.

Code to Finetune:

# Create a MirroredStrategy.
import tensorflow as tf
from tensorflow import keras

def f1_m(y_true, y_pred):
    precision = precision_m(y_true, y_pred)
    recall = recall_m(y_true, y_pred)
    return 2*((precision*recall)/(precision+recall+K.epsilon()))


def recall_m(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    recall = true_positives / (possible_positives + K.epsilon())
adam = tf.keras.optimizers.Adam(learning_rate=0.00005, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)
    return recall

def precision_m(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    return precision
dependencies = {
    'f1_m': f1_m,
    'recall_m': recall_m,
    'precision_m': precision_m,

}
strategy = tf.distribute.MirroredStrategy()
print("Number of devices: {}".format(strategy.num_replicas_in_sync))
# Open a strategy scope.
with strategy.scope():
    adam = tf.keras.optimizers.Adam(learning_rate=0.00005, amsgrad=True, name='Adam',)
    model = keras.models.load_model('/home/xylexa/Desktop/normal_abnormal/final experiments/10000_sample/finetune/best_resnet50_finetune_10000_sample.h5', custom_objects=dependencies)
    model.summary()

# Let's take a look to see how many layers are in the base model
print("Total number of layers in the Base model: ", len(model.layers))
print("Total number of layers to be Train in Base model: ",len(model.trainable_variables))

any updates and suggestions will be appreciated. :)

Answer 2

You have to compile the model after unfreezing and then start training, no need for saving it to disk before.