NaN for Keras Tuner score for RandomSearch

Question

I am trying out Keras (2.8.0) autotuner for a regression problem. Here is my code:

import pandas as pd
from tensorflow import keras
from keras import layers, losses
from keras_tuner.tuners import RandomSearch

df = pd.read_csv('./datasets/data.csv')

X = df.iloc[:, :-1]
y = df.iloc[:, -1]


def build_model(hp):
    model = keras.Sequential()
    for i in range(hp.Int('num_layers', 2, 20)):
        model.add(layers.Dense(units=hp.Int('units_' + str(i),
                                            min_value=32,
                                            max_value=512,
                                            step=32),
                               activation='relu'))
    model.add(layers.Dense(1, activation='linear'))
    model.compile(
        optimizer=keras.optimizers.Adam(
            hp.Choice('learning_rate', [1e-2, 1e-3, 1e-4])),
        loss=losses.MeanAbsoluteError(reduction="auto", name="mean_absolute_error"),
        metrics=['mean_absolute_error'])
    return model


tuner = RandomSearch(
    build_model,
    objective='val_mean_absolute_error',
    max_trials=5,
    executions_per_trial=3,
    directory='project',
    project_name='TunerTest')

model_summary = tuner.search_space_summary()

print(model_summary)

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

tuner.search(X_train, y_train,
             epochs=5,
             validation_data=(X_test, y_test))

tuner.results_summary()

Code runs without any errors (so to assume) but in the results, for all the iterations, the Score is nan. Below is the output of one of the runs:

Showing 10 best trials
<keras_tuner.engine.objective.Objective object at 0x7fef945235d0>
Trial summary
Hyperparameters:
num_layers: 12
units_0: 128
units_1: 224
learning_rate: 0.01
units_2: 32
units_3: 32
units_4: 32
units_5: 32
units_6: 32
units_7: 32
units_8: 32
units_9: 32
units_10: 32
units_11: 32
Score: nan

What could I be doing wrong here?

Answer 1

It might be due to your data having nan values or values very close to zero. Another issue could be the variance in your data. Maybe try normalizing it to avoid exploding gradients or even applying gradient clipping. When using random data, your code works and the result is not nan:

import pandas as pd
from tensorflow import keras
from keras import layers, losses
from keras_tuner.tuners import RandomSearch
import numpy as np

X = np.random.random((500, 5))
y = np.random.random((500, 1))

def build_model(hp):
    model = keras.Sequential()
    for i in range(hp.Int('num_layers', 2, 20)):
        model.add(layers.Dense(units=hp.Int('units_' + str(i),
                                            min_value=32,
                                            max_value=512,
                                            step=32),
                               activation='relu'))
    model.add(layers.Dense(1, activation='linear'))
    model.compile(
        optimizer=keras.optimizers.Adam(
            hp.Choice('learning_rate', [1e-2, 1e-3, 1e-4])),
        loss=losses.MeanAbsoluteError(reduction="auto", name="mean_absolute_error"),
        metrics=['mean_absolute_error'])
    return model


tuner = RandomSearch(
    build_model,
    objective='val_mean_absolute_error',
    max_trials=5,
    executions_per_trial=3,
    directory='project',
    project_name='TunerTest')

model_summary = tuner.search_space_summary()

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

tuner.search(X_train, y_train,
             epochs=5,
             validation_data=(X_test, y_test))

tuner.results_summary()

...
Trial summary
Hyperparameters:
num_layers: 20
units_0: 512
units_1: 384
learning_rate: 0.01
units_2: 32
units_3: 32
units_4: 32
units_5: 32
units_6: 32
units_7: 32
units_8: 32
units_9: 32
units_10: 32
units_11: 32
units_12: 32
units_13: 32
units_14: 32
units_15: 32
units_16: 32
units_17: 32
units_18: 32
units_19: 32
Score: 0.24690431356430054

Answer 2

If I am not wrong this was an example from Krish Naik's Dl videos; I tried the exact same code and encountered the following issue. I just replaced the null values in the dataset and it worked fine.