If your spark jobs are mostly batch workloads, that can tolerate moderately infrequent failures and restarts, try using google dataproc with preemptible vms or amazon emr using spot instances.<p>Depending on your use case, you might spend many times less than you would using regular VMs. Many instances that are several dollars an hour on AWS can be used for a fraction of the price.<p>Its also fairly easy to automate the region selection and bid (on AWS that is, not sure about gcloud).<p>If you need streaming, obviously this might not be the way to go.
Check out AWS Glue: <a href="https://aws.amazon.com/glue/" rel="nofollow">https://aws.amazon.com/glue/</a><p>Disclosure: I work on this service
All 3 major cloud providers have offerings in this space. Amazon [0], Google [1], Microsoft [2].<p>[0]: <a href="https://aws.amazon.com/emr/" rel="nofollow">https://aws.amazon.com/emr/</a><p>[1]: <a href="https://cloud.google.com/dataproc/" rel="nofollow">https://cloud.google.com/dataproc/</a><p>[2]: <a href="https://azure.microsoft.com/en-us/services/databricks/" rel="nofollow">https://azure.microsoft.com/en-us/services/databricks/</a>
Run Spark on a managed Kubernetes like GKE? There is experimental support for using Kubernetes as the cluster manager.<p><a href="https://apache-spark-on-k8s.github.io/userdocs/index.html" rel="nofollow">https://apache-spark-on-k8s.github.io/userdocs/index.html</a>
You can try Qubole [0]. The pricing is a small percentage of what you pay to the cloud provider, so it's predictable to an extent.<p>[0]: <a href="https://www.qubole.com/" rel="nofollow">https://www.qubole.com/</a><p>Disclosure: I work here.
We're using Spark on EMR with Data Pipeline to do ETL and to run Scheduled Jobs. Data pipelines terminates the cluster once ETL or job gets completed, helps us a lot to save cost.