Configurate Spark by given Cluster
<p>I have to send some applications in python to a Apache Spark cluster. There is given a Clustermanager and some worker nodes with the addresses to send the Application to.</p>

<p>My question is, how to setup and to configure Spark on my local computer to send those requests with the data to be worked out to the cluster?</p>

<p>I am working on Ubuntu 16.xx and already installed java and scala. I have searched the inet but the most find is how to build the cluster or some old advices how to do it, which are out of date.</p>

<p>your question is unclear. If the data are on your local machine, you should first copy your data to the cluster on HDFS filesystem. Spark can works in 3 modes with YARN (are u using YARN or MESOS ?): cluster, client and standalone. What you are looking for is client-mode or cluster mode. But if you want to start the application from your local machine, use client-mode. If you have an SSH access, you are free to use both.</p>

<p>The simplest way is to copy your code directly on the cluster if it is properly configured then start the application with the <code>./spark-submit</code> script, providing the class to use as an argument. It works with python script and java/scala classes (I only use python so I don't really know)</p>

java python scala apache-spark pyspark