Is Spark's KMeans unable to handle bigdata?
<p>KMeans has several parameters for its <a href="http://spark.apache.org/docs/latest/api/python/pyspark.mllib.html?highlight=kmeans#pyspark.mllib.clustering.KMeans.train" rel="nofollow">training</a>, with initialization mode is defaulted to kmeans||. The problem is that it marches quickly (less than 10min) to the first 13 stages, but then <strong>hangs completely</strong>, without yielding an error!</p>

<p><strong>Minimal Example</strong> which reproduces the issue (it will succeed if I use 1000 points or random initialization):</p>

<pre><code>from pyspark.context import SparkContext

from pyspark.mllib.clustering import KMeans
from pyspark.mllib.random import RandomRDDs


if __name__ == "__main__":
    sc = SparkContext(appName='kmeansMinimalExample')

    # same with 10000 points
    data = RandomRDDs.uniformVectorRDD(sc, 10000000, 64)
    C = KMeans.train(data, 8192,  maxIterations=10)    

    sc.stop()
</code></pre>

<p>The job does nothing (it doesn't succeed, fail or progress..), as shown below. There are no active/failed tasks in the Executors tab. Stdout and Stderr Logs don't have anything particularly interesting:</p>

<p><a href="http://i.stack.imgur.com/aggpL.png" rel="nofollow"><img src="http://i.stack.imgur.com/aggpL.png" alt="enter image description here"></a></p>

<p>If I use <code>k=81</code>, instead of 8192, it will succeed:</p>

<p><a href="http://i.stack.imgur.com/zBKqG.png" rel="nofollow"><img src="http://i.stack.imgur.com/zBKqG.png" alt="enter image description here"></a></p>

<p>Notice that the two calls of <code>takeSample()</code>, <a href="http://stackoverflow.com/questions/38986395/sparkkmeans-calls-takesample-twice">should not be an issue</a>, since there were called twice in the random initialization case.</p>

<p>So, what is happening? Is Spark's Kmeans <strong>unable to scale</strong>? Does anybody know? Can you reproduce?</p>

<hr>

<p>If it was a memory issue, <a href="https://gsamaras.wordpress.com/code/memoryoverhead-issue-in-spark/" rel="nofollow">I would get warnings and errors, as I had been before</a>. </p>

<p>Note: placeybordeaux's comments are based on the execution of the job in <em>client mode</em>, where the driver's configurations are invalidated, causing the exit code 143 and such (see edit history), not in cluster mode, where there is <em>no error reported at all</em>, the application <em>just hangs</em>.</p>

<hr>

<p>From zero323: <a href="http://stackoverflow.com/questions/35512139/why-is-spark-mllib-kmeans-algorithm-extremely-slow">Why is Spark Mllib KMeans algorithm extremely slow?</a> is related, but I think he witnesses some progress, while mine hangs, I did leave a comment...</p>

<p><a href="http://i.stack.imgur.com/osDrR.png" rel="nofollow"><img src="http://i.stack.imgur.com/osDrR.png" alt="enter image description here"></a></p>

<p>I think the 'hanging' is because your executors keep dying. As I mentioned in a side conversation, this code runs fine for me, locally and on a cluster, in Pyspark and Scala. However, it takes a lot longer than it should. It is almost all time spent in k-means|| initialization.</p>

<p>I opened <a href="https://issues.apache.org/jira/browse/SPARK-17389" rel="nofollow">https://issues.apache.org/jira/browse/SPARK-17389</a> to track two main improvements, one of which you can use now. Edit: really, see also <a href="https://issues.apache.org/jira/browse/SPARK-11560" rel="nofollow">https://issues.apache.org/jira/browse/SPARK-11560</a></p>

<p>First, there are some code optimizations that would speed up the init by about 13%.</p>

<p>However most of the issue is that it default to 5 steps of k-means|| init, when it seems that 2 is almost always just as good. You can set initialization steps to 2 to see a speedup, especially in the stage that's hanging now.</p>

<p>In my (smaller) test on my laptop, init time went from 5:54 to 1:41 with both changes, mostly due to setting init steps.</p>

python apache-spark bigdata k-means apache-spark-mllib