Setup Apache Sedona on EMR

Question

I want to be able to use Apache Sedona for distributed GIS computing on AWS EMR. We need the right bootstrap script to have all dependencies.

I tried setting up Geospark using EMR 5.33 using the Jars listed here. It didn't work as some dependencies were still missing.

I then manually set Sedona up on local, found the difference of Jars between Spark 3 and the Sedona setup and came up with following bootstrap script

#!/bin/bash
sudo pip3 install numpy
sudo pip3 install boto3 pandas findspark shapely py4j attrs
sudo pip3 install geospark --no-dependencies
sudo pip3 install apache-sedona
sudo aws s3 cp s3://emr_setup/apache-sedona-1.0.1-incubating-bin/sedona-python-adapter-2.4_2.11-1.0.1-incubating.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/apache-sedona-1.0.1-incubating-bin/sedona-viz-2.4_2.11-1.0.1-incubating.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/geospark_bin/postgresql-42.2.23.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/sedona-core-2.4_2.11-1.0.1-incubating.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/stream-2.7.0.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/orc-core-1.5.5-nohive.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/jersey-media-jaxb-2.22.2.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/hadoop-mapreduce-client-common-2.6.5.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/hadoop-mapreduce-client-shuffle-2.6.5.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/org.w3.xlink-24.0.jar /usr/lib/spark/jars/
sudo aws s3 cp s3:///emr_setup/spark_2.4_2.11_sedona_all_jars/minlog-1.3.0.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/jersey-client-2.22.2.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/xz-1.5.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/pyrolite-4.13.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/hadoop-yarn-common-2.6.5.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/curator-recipes-2.6.0.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/aopalliance-1.0.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/commons-configuration-1.6.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/commons-beanutils-1.7.0.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/gt-metadata-24.0.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/spark-unsafe_2.11-2.4.7.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/objenesis-2.5.1.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/commons-httpclient-3.1.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/stax-api-1.0-2.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/hk2-api-2.4.0-b34.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/apacheds-i18n-2.0.0-M15.jar /usr/lib/spark/jars/

The EMR setup starts, but the attached notebooks to the script don't seem to be able to start. The master seems to fail for some reason.

Need help with preparing the right bootstrap script to install Apache Sedona on EMR 6.0.