Configurate Spark by given Cluster
<p>I have to send some applications in python to a Apache Spark cluster. There is given a Clustermanager and some worker nodes with the addresses to send the Application to.</p>

<p>My question is, how to setup and to configure Spark on my local computer to send those requests with the data to be worked out to the cluster?</p>

<p>I am working on Ubuntu 16.xx and already installed java and scala. I have searched the inet but the most find is how to build the cluster or some old advices how to do it, which are out of date.</p>

<p>i assume you remote cluster is running and you are able to submit jobs on it from remote server itself. what you need is ssh tuneling. Keep in mind that it does not work with aws.</p>

<pre><code>ssh -f user@personal-server.com -L 2000:personal-server.com:7077 -N
</code></pre>

<p>read more here: <a href="http://www.revsys.com/writings/quicktips/ssh-tunnel.html" rel="nofollow">http://www.revsys.com/writings/quicktips/ssh-tunnel.html</a></p>

java python scala apache-spark pyspark