Clustering dense data points horizontally
<p>I have a set of about 34,000 data labels with their respective features (state probabilities) in a 2D numpy array that, visualised as a scatter plot, looks <img src="http://i.stack.imgur.com/UlYJG.png" alt="like this">.</p>

<p>It's easy to see that the majority of b data points is at the bottom and quite dense. I would like to use a clustering algorithm to extract the bottom area. I don't strive for a perfect result. This is simply about extracting the majority of b points.</p>

<p>So far I have tried the DBSCAN algorithm:</p>

<pre><code>import sklearn.cluster as sklc

data1, data2 = zip(*dist_list[1])
data = np.array([data1, data2]).T

core_samples, labels_db = sklc.dbscan(
    data,  # array has to be (n_samples, n_features)
    eps=2.0,
    min_samples=5,
    metric='euclidean',
    algorithm='auto'
)
core_samples_mask = np.zeros_like(labels_db, dtype=bool)
core_samples_mask[core_samples] = True

unique_labels = set(labels_db)

n_clusters_ = len(unique_labels) - (1 if -1 in labels_db else 0)

colors = plt.cm.Spectral(np.linspace(0, 1, len(unique_labels)))
for k, col in zip(unique_labels, colors):
    if k == -1:
        # Black used for noise.
        col = 'k'

    class_member_mask = (labels_db == k)

    xy = data[class_member_mask &amp; core_samples_mask]
    plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=col, markeredgecolor='k', markersize=6)

    xy = data[class_member_mask &amp; ~core_samples_mask]
    plt.plot(xy[:, 0], xy[:, 1], 'x', markerfacecolor=col, markeredgecolor='k', markersize=4)

plt.rcParams["figure.figsize"] = (15, 15)
plt.title('Estimated number of clusters: %d' % n_clusters_)
plt.show()
</code></pre>

<p>which yields <img src="http://i.stack.imgur.com/PPX1E.png" alt="this plot">.</p>

<p>Increasing the minimal amount of samples simply results in smaller vertical lines being classified as noise and the longer (and denser) vertical lines staying.</p>

<p>I also tried clustering with <code>scipy.cluster.hierarchy</code>:</p>

<pre><code>thresh = 2
clusters = hcluster.fclusterdata(data, thresh, criterion="distance")

plt.scatter(*data.T, c=clusters)
title = "t=%f, n=%d" % (thresh, len(set(clusters)))
plt.title(title)
plt.show()
</code></pre>

<p>which resulted in a similar vertical classification. Please view the comment for the plot. With my reputation I'm not allowed to post more than two links yet.</p>

<p>Now my question is, did I make a mistake in the calibration of the algorithms? Or was my choice of algorithms wrong in the first place? How can I extract the dense area of b data points?</p>

<ol>
<li><p>Never include ID attributes. Your "node index" supposedly should not be used for similarity computations, should it?</p></li>
<li><p>When your <strong>attributes have different units and scale</strong> you need to be very careful. A popular heuristic is <code>StandardScaler</code>, i.e. normalize each attribute to have unit variance. But given the distribution of your data (uniform on x, skewed on y) I do not think the use of standardization is well supported by theory - it assumes Gaussian distributions.</p></li>
</ol>

<p>The combination of these two is why you get this result: your x axis completely dominates the result. It literally has over 10000x the influence than your y axis  You are effectively clustering only based on Node ID...</p>

<p><strong>Never assume clustering "just works"</strong>. All of these methods are <em>very</em> sensitive to preprocessing, and therefor need to be applied with care.</p>

python scipy scikit-learn cluster-analysis dbscan