Spark History server not listing completed jars

Question

I'm running Spark standalone jobs in Windows. I would like to monitor my Spark jobs using the spark history server. I have launched spark history server with below command and history server is successfully launched.

C:\Users\Documents\Prev_D_Folder\Softwares\Softwares\spark-2.2.0-bin-hadoop2.7\spark-2.2.0-bin-hadoop2.7\spark-2.2.0-bin-hadoop2.7\bin>spark-class.cmd org.apache.spark.deploy.history.HistoryServer

I ran spark job locally but history server does not list any completed jobs.

My spark spark-defaults.conf file content is:

spark.eventLog.enabled           true
spark.eventLog.dir               file:///C:/Users/m1047320/Documents/logs
spark.history.provider = org.apache.spark.deploy.history.FsHistoryProvider
spark.history.fs.logDirectory   file:///C:/Users/m1047320/Documents/logs-hist

I have read, write and execute permissions for both log directories.

Answer 1

The directory for spark.eventLog.dir and spark.history.fs.logDirectory should be the same.

From the documentation, spark.eventLog.dir is the log base directory:

Base directory in which Spark events are logged, if spark.eventLog.enabled is true. Within this base directory, Spark creates a sub-directory for each application, and logs the events specific to the application in this directory [...]

While spark.history.fs.logDirectory configures the directory the history server uses to read logs:

For the filesystem history provider, the URL to the directory containing application event logs to load.

The reason you can't find any completed jobs is due to the history server reading an empty directory.

Answer 2

all the settings posted in question remain the same along with what I need to add -Dspark.eventLog.enabled=true -Dspark.history.fs.logDirectory=file:///C:/sparkhistorylogs -Dspark.eventLog.dir=file:///C:/sparkhistorylogs

while submitting spark application or configure in Vmoptions in intelliji.

I missed above step hence history server was not showing completed jobs. Now i am able access completed job's info through history server at https://localhost:18080