'Unable to register with external shuffle server. Failed to connect on standalone Spark cluster

I followed the Dynamic allocation setup configuration however, getting the following error when starting the executors.

ERROR TaskSchedulerImpl: Lost executor 0 on 192.168.0.167: Unable to create executor due to Unable to register with external shuffle server due to : Failed to connect to /192.168.0.167:7337 20/03/25 13:49:01 WARN TransportChannelHandler: Exception in connection from /192.168.0.167:55155 java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)

My Spark config on standalone cluster (local machine):

spark = SparkSession \
    .builder \
    .appName("DynanicAlloc")\
    .config("spark.master","spark://localhost:7077")\
    .config("spark.ui.port", 8050)\
    .config("spark.driver.maxResultSize", 0)\
    .config('spark.rpc.message.maxSize','1024')\
    .config("spark.dynamicAllocation.enabled", True)\
    .config("spark.shuffle.service.enabled", True)\
    .config("spark.dynamicAllocation.initialExecutors", "1")\
    .config("spark.dynamicAllocation.executorIdleTimeout", "120s")\
    .config("spark.dynamicAllocation.schedulerBacklogTimeout", "1s")\
    .config("spark.executor.cores", "1")\
    .config("spark.executor.memory", "512m")\
    .config("spark.rdd.compress", "true")\
    .config('spark.scheduler.mode','FAIR')\
    .config("spark.sql.execution.arrow.enabled", "true")\
    .getOrCreate()

apache-spark pyspark

Solution 1:^[1]

Remove configs related to dynamicAllocation and try. Standalone mode doesnot require those configs.

spark = SparkSession \
    .builder \
    .appName("DynanicAlloc")\
    .config("spark.master","spark://localhost:7077")\
    .config("spark.ui.port", 8050)\
    .config("spark.driver.maxResultSize", 0)\
    .config('spark.rpc.message.maxSize','1024')\
    .config("spark.executor.cores", "1")\
    .config("spark.executor.memory", "512m")\
    .config("spark.rdd.compress", "true")\
    .config('spark.scheduler.mode','FAIR')\
    .config("spark.sql.execution.arrow.enabled", "true")\
    .getOrCreate()

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	Sunil

'Unable to register with external shuffle server. Failed to connect on standalone Spark cluster

Solution 1:[1]

Sources

Related Questions

Solution 1:^[1]