Clustering text documents and obtaining duplicate top terms
<p>I found the code in <a href="http://stackoverflow.com/questions/27889873/clustering-text-documents-using-scikit-learn-kmeans-in-python?rq=1">this post</a> to be very helpful. (I would add a comment to that post but I need 50 reputation points.)</p>

<p>I used the same code in the post above but added a test document I have been using for debugging my own clustering code. For some reason a word in 1 document appears in both clusters. </p>

<p>The code is:</p>

<p>Update: I added "Unique sentence" to the documents below. </p>

<pre><code>documents = ["I ran yesterday.", 
                "The sun was hot.", 
                "I ran yesterday in the hot and humid sun.", 
                "Yesterday the sun was hot.",
                "Yesterday I ran in the hot sun.",
                "Unique sentence." ]

vectorizer = TfidfVectorizer(stop_words='english')
X = vectorizer.fit_transform(documents)

#cluster documents    
true_k = 2
model = KMeans(n_clusters=true_k, init='k-means++', max_iter=100, n_init=1)
model.fit(X)

#print top terms per cluster clusters    
print("Top terms per cluster:")
order_centroids = model.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(true_k):
    print ("Cluster %d:" % i,)
    for ind in order_centroids[i, :10]:
        print(' %s' % terms[ind])
    print
</code></pre>

<p>The output I receive is:</p>

<p>UPDATE: I updated the output below to reflect the "unique sentence" above. </p>

<p>Cluster 0:
 sun
 hot
 yesterday
 ran
 humid
 unique
 sentence
Cluster 1:
 unique
 sentence
 yesterday
 sun
 ran
 humid
 hot</p>

<p>You'll note that "humid" appears as a top term in both clusters even though it's in just 1 line of the documents above. I would expect a unique word, like "humid" in this case, to be a top term in just 1 of the clusters. </p>

<p>Thanks! </p>

<p>Why would clusters have distinct top terms?</p>

<p>Consider the clustering worked (very often it doesn't - beware), would you consider these clusters to be bad or good:</p>

<ul>
<li>banana fruit</li>
<li>apple fruit</li>
<li>apple computer</li>
<li>windows computer</li>
<li>window blinds</li>
</ul>

<p>If I would ever get such clusters, I wouldmbe happy (iknfact, I would believe I am seeing an error, because these are much too god results. Text clustering is always borderline to non-working).</p>

<p><strong>With text clusters, it's a lot about word combinations, not just single words</strong>. Apple fruit and apple computer are not the same.</p>

python scikit-learn cluster-analysis k-means