efficient filtering near/inside clusters after they found - python
<p>essentially I applied a <code>DBSCAN</code> algorithm (<code>sklearn</code>) with an euclidean distance on a subset of my original data. I found my clusters and all is fine: except for the fact that I want to keep only values that are far enough from those on which I did not run my analysis on. I have a new distance to test such new stuff with and I wanted to understand how to do it <strong>WITHOUT</strong> numerous nested loops.</p>

<p>in a picture:</p>

<p><a href="http://i.stack.imgur.com/ugebW.png" rel="nofollow"><img src="http://i.stack.imgur.com/ugebW.png" alt="enter image description here"></a></p>

<p>my found clusters are in blue whereas the red ones are the points to which I don't want to be near. the crosses are the points belonging to the cluster that are carved out as they are within the new distance I specified.</p>

<p>now, as much I could do something of the sort:</p>

<pre><code>for i in red_points:
    for j in blu_points:
        if dist(i,j) &lt; given_dist:
            original_dataframe.remove(j)
</code></pre>

<p>I refuse to believe there isn't a vectorized method. also, I can't afford to do as above simply because I'll have huge tables to operate upon and I'd like to avoid my CPU to evaporate away.</p>

<p>any and all suggestions welcome</p>

<p>If you need exact answers, the fastest implementation should be sklearn's pairwise distance calculator:
<a href="http://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.pairwise_distances.html" rel="nofollow">http://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.pairwise_distances.html</a></p>

<p>If you can accept an approximate answer, you can do better with the kd tree's queryradius(): <a href="http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KDTree.html" rel="nofollow">http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KDTree.html</a></p>

python scikit-learn cluster-computing dbscan