'Spark in SBT console: "Could not find spark-version-info.properties"

I'm trying to instantiate a SparkContext inside a SBT console, using the following scala commands:

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext.*

val sc: SparkContext = new SparkContext(new SparkConf().setAppName("wikipedia").setMaster("local[*]")) 

...but I get the following error: Could not find spark-version-info.properties. (Note that when I use sbt run to execute code, everything works fine, the error only occurs in the interactive sbt console mode):

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties org.apache.spark.SparkException: Could not find spark-version-info.properties at org.apache.spark.package$SparkBuildInfo$.(package.scala:62)
at org.apache.spark.package$.(package.scala:93) at org.apache.spark.SparkContext.$anonfun$new$1(SparkContext.scala:193)
at org.apache.spark.internal.Logging.logInfo(Logging.scala:57) at org.apache.spark.internal.Logging.logInfo$(Logging.scala:56) at org.apache.spark.SparkContext.logInfo(SparkContext.scala:82) at org.apache.spark.SparkContext.(SparkContext.scala:193) ... 62 elided

My build.sbt file:

scalaVersion := "3.1.1"
scalacOptions ++= Seq("-language:implicitConversions", "-deprecation")
libraryDependencies ++= Seq(
  "org.scalameta" %% "munit" % "0.7.26" % Test,
  ("org.apache.spark" %% "spark-core" % "3.2.1").cross(CrossVersion.for3Use2_13),
  ("org.apache.spark" %% "spark-sql" % "3.2.1").cross(CrossVersion.for3Use2_13)
)

It would be great if somebody could help me with this issue.

My system configuration:

  • Ubuntu 18.04
  • Java version: openjdk "1.8.0_312"
  • scala version: 3.1.1
  • spark version: 3.2.1

I installed sbt/scala using the following command (as recommended on the official website):

curl -fL https://github.com/coursier/launchers/raw/master/cs-x86_64-pc-linux.gz | gzip -d > cs && chmod +x cs && ./cs setup



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source