Dataproc secondary workers not used

Question

I've got a Dataproc cluster going on configured this way:

{
    "worker_config": {
        "num_instances": 20
    },
    "secondary_worker_config": {
        "num_instances": 10,
        "is_preemptible": True
    }
    # no autoscaling set
}

I've omitted on purpose master node details, machines types etc.

The problem is that at run time, Dataproc doesn't seem to use the secondary nodes at all:

• Secondary nodes don't get a green mark in the VM list of the cluster
• The sum of the available+allocated memory does not include the secondary nodes
• The number is the Yarn node managers is 20 which is the number of primary nodes only

More importantly: the underlying Spark job execution time is comparable with a 20 machines cluster and I see no benefits from using secondary preemptible nodes.

Thank you!

Answer 1

In general, secondary worker nodes are created through the managed instance group. If you have secondary workers attached to your cluster and it's not activated(not running) in this case, there is possibility of permission issues in the service account.

Refer the following article for the detailed troubleshooting guidelines https://cloud.google.com/dataproc/docs/concepts/compute/secondary-vms#troubleshooting_secondary_workers