Execute Jobs using Spark + Cassandra utilizing data locality
<p>I have an exec, which accepts a cassandra primary key as input.</p>

<pre><code>Cassandra Row: (id, date), clustering_key, data
./exec id date
</code></pre>

<p>Each exec can access multiple rows for given primary key. After performing execution on data, it stores results in a DB.</p>

<p>I have multiple such execs and I want to run exec on a node which stores data. How can I achieve this using spark?</p>

<p>Also, how do I receive node ip in which the exec has run [For verification purposes].</p>

<p>Note: In exec I am accessing data by executing query:</p>

<pre><code>select data from table where id = t_id and date = t_date and clustering_key = t_clustering_key
</code></pre>

<p>If you want to use Spark (with data locality), you have to write Spark program to do the same thing <code>exec</code> is doing. Spark driver (you can use DataStax Cassandra/Spark Connector) takes care of locality issues automatically. </p>

<p>If you want to exploit the data locality without writing Spark program, then it's going to be difficult and I don't know if you need Spark at all, in that case.</p>

<p>P.S. If you are doing a shuffle operation in Spark (which I don't think you are doing), then writing a Spark program is also not going to help with data locality.</p>

<p>References:
<a href="http://events.linuxfoundation.org/sites/events/files/slides/Spark-Cassandra%20Integration,%20theory%20and%20practice%20ApacheBigData%202015_0.pdf?bcsi_scan_fd86d3dd427d821e=0&amp;bcsi_scan_filename=Spark-Cassandra%20Integration,%20theory%20and%20practice%20ApacheBigData%202015_0.pdf" rel="nofollow">Presentation by Datastax employee about Spark and Cassandra data locality</a></p>

python apache-spark cassandra