.persist() line sometimes leads to Java Out of Heap Space error
<p>As far as I know, when you use <code>.persist()</code>, writing the line <code>persist</code> sets only the persistence level, and then the next <code>action</code> in the script will cause the actual persistence work to be invoked.</p>

<p>However, sometimes, seemingly depending on the dataframe, <code>persist()</code> will lead to a Java out of heap space error.</p>

<p>What is the intended behavior of persist, and why could this simple line actually lead to this memory error?</p>

<p>The whole point of <a href="http://spark.apache.org/docs/latest/programming-guide.html#rdd-persistence" rel="nofollow">RDD Persistence</a> is to store intermediate results in memory, allowing faster access on subsequent use. There are several different levels of persistence, ranging from <code>MEMORY_ONLY</code> (the default), over <code>MEMORY_AND_DISK</code>, up to <code>DISK_ONLY</code>. Persisting purely to memory means that there has to be enough heap space for the persist to work. If you run out of heap memory you can</p>

<ul>
<li>go for a lower persistence level, </li>
<li>reduce the size of your partitions,</li>
<li>reduce the total number of persisted RDD stages, e.g., by <code>unpersist</code>ing them. </li>
</ul>

<p>Finding the right balance is one of the key challenges in Spark to achieve a good tradeoff between memory and CPU usage.</p>

python apache-spark pyspark elastic-map-reduce