Nearest neighbour in pyspark using euclidean distance or similar
<p>So I need to find nearest neighbors of a given row in pyspark DF using euclidean distance or anything. the data that I have 20+ columns, more than thousand rows and all the values are numbers.</p>

<p>I am trying to oversample some data in pyspark, as mllib doesn't have inbuilt support for it, i decided to create it myself using smote.</p>

<p>my approach till now has been to convert all the categorical distance into index using <strong>stringtoindex</strong> so that i can find the euclidean distance and neighbors and hence perform smote.</p>

<p>I am fairly new to spark and ml. Any help would be appreciated.</p>

<p>Not tried but Ive found this script: <a href="https://github.com/jakac/spark-python-knn/blob/master/python/gaussalgo/knn/knn.py" rel="nofollow">https://github.com/jakac/spark-python-knn/blob/master/python/gaussalgo/knn/knn.py</a></p>

<p>If your data are dataframe, you should first merge your column into a vector with vectorASsembler <a href="https://spark.apache.org/docs/latest/ml-features.html#vectorassembler" rel="nofollow">https://spark.apache.org/docs/latest/ml-features.html#vectorassembler</a>, then use <code>df.select("id", "yourColumnVector")</code></p>

<p>The library I provided seems to work only with rdd, so you should convert your dataframe to RDD using <code>df.rdd()</code> or something</p>

python pyspark nearest-neighbor knn