How to Setup Adaptive Learning Rate in Keras

Question

Below is my code:

model = Sequential([
    Dense(32, input_shape=(32,), activation = 'relu'),
    Dense(100, activation='relu'),
    Dense(65, input_shape=(65,), activation='softmax')
])

model.summary()
model.compile(SGD(lr=.1), loss='binary_crossentropy', metrics=['accuracy'])
model.fit(train_samples, train_labels, batch_size=1000, epochs=1000,shuffle = True, verbose=2)

How will I set an adaptive learning rate of my model?

Answer 1

You don't need to recompile the model as the other answer suggested. Keras comes with callbacks which can be used for this task. More precisely, you can use LearningRateScheduler callback and pass it some function that will adapt the learning rate based on the current epoch index.

Suppose that you want your learning rate to be some number times the epoch index (probably not the best idea but easy to comprehend)

def adapt_learning_rate(epoch):
    return 0.001 * epoch

Now that we have our function we can create a learning scheduler that is responsible for calculating the learning rate at the beginning of each epoch.

my_lr_scheduler = keras.callbacks.LearningRateScheduler(adapt_learning_rate)

Last thing to do is to pass this callback to the fit method.

model.fit(X, y, ..., callbacks=[my_lr_scheduler])

Answer 2

You may use a workaround.

For each_iteration in range(0, MaxEpoch):

1. Specify your own learning rate function that outputs a learning rate lr with respect to per epoch. The lr is then passed to your_optimiser

2. run model.compile(...optimizer=your_optimiser...)

3. run model.fit(...epochs = 1...)

4. After the ONE epoch, use model.save_weights(...)

5. Load weights by model.load_weights(...) for next iteration. See here for details https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model

In fact, #4 and #5 enables you to do a transfer learning

Answer 3

You need to replace SGD here

model.compile(SGD(lr=.1), loss='binary_crossentropy', metrics=['accuracy'])

with one of the provided optimizers, e.g. Adam:

model.compile(Adam(lr=.1), loss='binary_crossentropy', metrics=['accuracy'])

Read this https://keras.io/optimizers/

Answer 4

you can just add a decay parameter to the optimiser (although that's not exactly adaptive, it assumes that the model will indeed converge as it learns), starting with a large learning rate and decreasing it gradually as the model converges. your compiler becomes:

model.compile(SGD(lr=.1, decay=1e-4), loss='binary_crossentropy', metrics=['accuracy'])