'Spark in SBT console: "Could not find spark-version-info.properties"
I'm trying to instantiate a SparkContext inside a SBT console, using the following scala commands:
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext.*
val sc: SparkContext = new SparkContext(new SparkConf().setAppName("wikipedia").setMaster("local[*]"))
...but I get the following error: Could not find spark-version-info.properties
. (Note that when I use sbt run
to execute code, everything works fine, the error only occurs in the interactive sbt console
mode):
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties org.apache.spark.SparkException: Could not find spark-version-info.properties at org.apache.spark.package$SparkBuildInfo$.(package.scala:62)
at org.apache.spark.package$.(package.scala:93) at org.apache.spark.SparkContext.$anonfun$new$1(SparkContext.scala:193)
at org.apache.spark.internal.Logging.logInfo(Logging.scala:57) at org.apache.spark.internal.Logging.logInfo$(Logging.scala:56) at org.apache.spark.SparkContext.logInfo(SparkContext.scala:82) at org.apache.spark.SparkContext.(SparkContext.scala:193) ... 62 elided
My build.sbt
file:
scalaVersion := "3.1.1"
scalacOptions ++= Seq("-language:implicitConversions", "-deprecation")
libraryDependencies ++= Seq(
"org.scalameta" %% "munit" % "0.7.26" % Test,
("org.apache.spark" %% "spark-core" % "3.2.1").cross(CrossVersion.for3Use2_13),
("org.apache.spark" %% "spark-sql" % "3.2.1").cross(CrossVersion.for3Use2_13)
)
It would be great if somebody could help me with this issue.
My system configuration:
- Ubuntu 18.04
- Java version: openjdk "1.8.0_312"
- scala version: 3.1.1
- spark version: 3.2.1
I installed sbt/scala using the following command (as recommended on the official website):
curl -fL https://github.com/coursier/launchers/raw/master/cs-x86_64-pc-linux.gz | gzip -d > cs && chmod +x cs && ./cs setup
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|