Bert Model Compile Error - TypeError: Invalid keyword argument(s) in `compile`: {'steps_per_execution'}

Question

I have been using bert and trying to compile the model using the below line of code.

model = TFBertForSequenceClassification.from_pretrained('bert-base-uncased')
optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate, epsilon=1e-08)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')

callbacks = [tf.keras.callbacks.ModelCheckpoint(filepath=OUTPUT_FOLDER+"\\bert_model.h5",
                                                        save_weights_only=True,
                                                        monitor='val_loss',
                                                        mode='min',
                                                        save_best_only=True)]

model.compile(optimizer=optimizer, loss=loss, metrics=[metric])

While compiling the code, I get the type error.

TypeError: Invalid keyword argument(s) in compile: {'steps_per_execution'}

The transformers package version im currently using is 4.11.3. The above code worked like a charm when I used tranformers 4.10.2

How do i get this to work with 4.11.3?

Answer 1

It's because your TensorFlow version is not up to date. I had the same issue a few hours ago. I updated my TensorFlow version to the newest version (and CUDA and cudnn to the respective version) and now it works fine. So just update your TensorFlow version and you will be fine.

Edit: I went from TensorFlow 2.3.0 to 2.7.0

Answer 2

This issue also arise when using an up to date Tensorlflow version, such as v2.8.0, and disabling eager execution.

tf.compat.v1.disable_eager_execution()

This happens because keras models rely on training_v1.py instead of training.py when using graph execution, and the former does not provide the steps_per_execution option.

The file training.py implements the base class for Keras models when using eager execution (default) and training_v1.py does the same for graph mode. While they intend to provide the same functionality, the option steps_per_execution is not available for graph mode though.