java.lang.NoClassDefFoundError: org/apache/log4j/spi/Filter in SparkSubmit

Question

I've been trying to submit applications to a Kubernetes. I have followed the tutorial in https://spark.apache.org/docs/latest/running-on-kubernetes.html such as building the spark image and etc.

But whenever I tried to run the command spark-submit, the pod always throw error. This is the logs from the command: kubectl logs <spark-driver-pods>:

Error: Unable to initialize main class org.apache.spark.deploy.SparkSubmit
Caused by: java.lang.NoClassDefFoundError: org/apache/log4j/spi/Filter

I have tried to use something like:

spark-submit
  ...
  --jars $(echo /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/pyspark/jars/*.jar | tr ' ' ',')
  ...

But that also still throw error.

Some notes related to my development environment:

• I use Kubernetes built-in the Docker desktop
• I use pyspark in conda environment, and yes I have activated the environment. That's why I can use pyspark in the terminal.

Anything else I should do? Or forget to do?

Answer 1

I'm using Maven, but I encountered this error while migrating from log4j 1.x to log4j 2.x and realized I still had some code that only worked with 1.x. Instead of refactoring code, I added this dependency to my pom.xml in order to maintain compatibility.

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-1.2-api</artifactId>
    <version>2.17.1</version>
</dependency>