For this second method we are going to submit the application using the rest api. the difference is we are going to put the main application file in an s3 bucket In order to do that we are going to run this request in a terminal:
curl -k -X POST \
'https://api.spotinst.io/ocean/spark/cluster/<your cluster id>/app?accountId=<your accountId>' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer ...' \
-d '{
"jobId": "spark-pi",
"configTemplateId": "ocean-workshop",
"configOverrides":
{
"type": "Python",
"sparkVersion": "3.2.0",
"mainApplicationFile": "s3://<your-bucket-name>/main.py",
"image": <replace-with-the-image-that-we-pushed-to-ecr>
}
}'