in TFF context : Is the evaluation step depends on training process?

Question

We all know that the evaluation step is quite important to evaluate our model on a test basis. I wanted to know if it is necessary to go through the round step(training) before doing the evaluation? that mean my code can be like this? it is correct like below??

emnist_train, emnist_test = tff.simulation.datasets.emnist.load_data()
...
def create_keras_model():
  return tf.keras.models.Sequential([
      tf.keras.layers.Input(shape=(784,)),
      tf.keras.layers.Dense(10, kernel_initializer='zeros'),
      tf.keras.layers.Softmax(),
  ])
def model_fn():
  # We _must_ create a new model here, and _not_ capture it from an external
  # scope. TFF will call this within different graph contexts.
  keras_model = create_keras_model()
  return tff.learning.from_keras_model(
      keras_model,
      dummy_batch=sample_batch,
      loss=tf.keras.losses.SparseCategoricalCrossentropy(),
      metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])
iterative_process = tff.learning.build_federated_averaging_process(
    model_fn,
    client_optimizer_fn=lambda: tf.keras.optimizers.SGD(learning_rate=0.02),
    server_optimizer_fn=lambda: tf.keras.optimizers.SGD(learning_rate=1.0))
state = iterative_process.initialize()
evaluation = tff.learning.build_federated_evaluation(model_fn)
train_metrics = evaluation(state.model, federated_train_data)
federated_test_data = make_federated_data(emnist_test, sample_clients)
test_metrics = evaluation(state.model, federated_test_data)

without going through this step

for round_num in range(2, 11):
  state, metrics = iterative_process.next(state, federated_train_data)
  print('round {:2d}, metrics={}'.format(round_num, metrics))

Is that possible and give me correct result? Thanks for you all

Answer 1

Its definitely possible to run a round of evaluation before training, but whether it is correct depends on the desired measurement.

The code above will likely report very low evaluation accuracy since the model isn't going through any train steps and is initialized to zero weight (assuredly not the best model parameters for the EMNIST dataset). I suspect this is not desired.

Moving the evaluation into the loop over rounds maybe informative, e.g. running an evaluation on the training data before taking local steps. This can help avoid some of the difficult interpretations of training metrics in Federated Learning. The code would look something like:

for round_num in range(2, 11):
  train_metrics = evaluation(state.model, federated_test_data)  # evaluate here
  state, _ = iterative_process.next(state, federated_train_data)  # ignore train metrics
  print('round {:2d}, metrics={}'.format(round_num, train_metrics))

test_metrics = evaluation(state.model, federated_test_data)