'Unable to start spark-shell failing to submit spark-submit

I am trying to submit spark-submit but its failing with as weird message.

 Error: Could not find or load main class org.apache.spark.launcher.Main
 /opt/spark/bin/spark-class: line 96: CMD: bad array subscript

this is the first time I am seeing this kind of error. I tried to check the code for the spark-class file but unable to decipher what is causing the issue.

# Turn off posix mode since it does not allow process substitution
set +o posix
CMD=()
DELIM=$'\n'
CMD_START_FLAG="false"
while IFS= read -d "$DELIM" -r ARG; do
  if [ "$CMD_START_FLAG" == "true" ]; then
    CMD+=("$ARG")
  else
    if [ "$ARG" == $'\0' ]; then
      # After NULL character is consumed, change the delimiter and consume command string.
      DELIM=''
      CMD_START_FLAG="true"
    elif [ "$ARG" != "" ]; then
      echo "$ARG"
    fi
  fi
done < <(build_command "$@")

COUNT=${#CMD[@]}
LAST=$((COUNT - 1))
LAUNCHER_EXIT_CODE=${CMD[$LAST]}

the line which is mentioned in the error message is

LAUNCHER_EXIT_CODE=${CMD[$LAST]}

Any pointer or any idea why the issue will help me a lot.

Thanks

Solution 1:^[1]

When I faced exactly the same problem, I looked in a bin directory of my Spark setup which is in your Windows PATH variable and when you run spark-submit <arguments> CMD searches for this command in Spark's bin. This directory can be found by executing echo %SPARK_HOME%\bin in CMD. And I saw two copies for every executable bin directory screenshot. This makes sense because we need different executable scripts for Windows and Linux. So finally I just typed spark-submit.cmd instead of spark-submit and everything worked like expected.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	LinFelix

'Unable to start spark-shell failing to submit spark-submit

Solution 1:[1]

Sources

Related Questions

Solution 1:^[1]