'Issue while running command spark-shell on windows
I have been trying to set up spark to use it further for pyspark library. I installed JDK, Hadoop and spark. Also provided the environment variables correctly. But I have been getting the below error after running the command spark-shell in windows command prompt.
'''Using Spark's default log4j profile: org/apache/spark/log4j- defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 22/05/06 18:51:58 WARN NativeCodeLoader: Unable to load native- hadoop library for your platform... using builtin-java classes where applicable java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0xc333c60) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0xc333c60 at org.apache.spark.storage.StorageUtils$. (StorageUtils.scala:213) at org.apache.spark.storage.StorageUtils$.(StorageUtils.scala)
at org.apache.spark.storage.BlockManagerMasterEndpoint.(BlockManagerMasterEndpoint.scala:110) at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348)
at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287) at org.apache.spark.SparkEnv$.create(SparkEnv.scala:336) at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191) at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)
at org.apache.spark.SparkContext.(SparkContext.scala:460) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943) at org.apache.spark.repl.Main$.createSparkSession(Main.scala:106)
... 51 elided :14: error: not found: value spark import spark.implicits._ ^ :14: error: not found: value spark import spark.sql ^'''
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|