How to manage a Apache Spark context in Django?
<p>I have a Django application that interacts with a Cassandra database and I want to try using Apache Spark to run operations on this database. I have some experience with Django and Cassandra but I'm new to Apache Spark.</p>

<p>I know that to interact with a Spark cluster first I need to create a SparkContext, something like this:</p>

<pre><code>from pyspark import SparkContext, SparkConf

conf = SparkConf().setAppName(appName).setMaster(master)
sc = SparkContext(conf=conf)
</code></pre>

<p>My question is the following: how should I treat this context? Should I instantiate it when my application starts and let it live during it's execution or should I start a SparkContext everytime before running an operation in the cluster and then kill it when the operation finishes? </p>

<p>Thank you in advance.</p>

<p>For the last days I've been working on this, since no one answered I will post what was my approach.</p>

<p>Apparently creating a SparkContext generates a bit of overhead, so stopping the context after every operation is not a good idea. </p>

<p>Also, there is no downfall, apparently, on letting the context live while the application runs.</p>

<p>Therefore, my approach was treating the SparkContext like a database connection, I created a singleton that instantiates the context when the application starts running and used it where needed.</p>

<p>I hope this can be helpful to someone, and I am open to new suggestions on how to deal with this, I'm still new to Apache Spark.</p>

python django apache apache-spark