Spark RDD S3 saveAsTextFile taking long time

Question

I have a Spark Streaming job on EMR which runs on batches of 30 mins, processes the data and finally writes the output to several different files in S3. Now the output step to S3 is taking too long (about 30mins) to write the files to S3. On investigating further, I found that the majority time taken is after all tasks have written the data in temporary folder (happens within 20s) and rest of the time taken is due to the fact that the master node is moving the S3 files from _temporary folder to destination folder and renaming them etc. (Similar to: Spark: long delay between jobs)

Some other details on the job configurations, file format etc are as below:

• EMR version: emr-5.22.0
• Hadoop version:Amazon 2.8.5
• Applications:Hive 2.3.4, Spark 2.4.0, Ganglia 3.7.2
• S3 files: Done using RDD saveAsTextFile API with S3A URL, S3 file format is text

Now although the EMRFS output committer is enabled by default in the job but it is not working since we are using RDDs and text file format which is supported post EMR 6.40 version only. One way that I can think of for optimizing the time taken in S3 save is by upgrading the EMR version, converting RDDs to DataFrames/Datasets and using their APIs instead of saveAsTextFile. Is there any other simpler solution possible to optimize the time taken for the job?

Answer 1

Is there any other simpler solution possible to optimize the time taken for the job?

unless you use an s3-specific committer, your jobs will not only be slow, they will be incorrect in the presence of failures. As this may matter to you,it is good that the slow job commits are providing an early warning of problems even before worker failures result in invalid output

options

1. upgrade. the committers were added for a reason.
2. use a real cluster fs (e.g HDFS) as the output then upload afterwards.

The s3a zero rename committers do work in saveAsTextFile, but they aren't supported by AWS and the ASF developers don't test on EMR as it is amazon's own fork. you might be able to get any s3a connector amazon ship to work, but you'd be on your own if it didn't.