doc2vec How to cluster DocvecsArray
<p>I've patched the following code from examples I've found over the web:</p>

<pre><code># gensim modules
from gensim import utils
from gensim.models.doc2vec import LabeledSentence
from gensim.models import Doc2Vec
from sklearn.cluster import KMeans

# random
from random import shuffle

# classifier

class LabeledLineSentence(object):
    def __init__(self, sources):
        self.sources = sources

        flipped = {}

        # make sure that keys are unique
        for key, value in sources.items():
            if value not in flipped:
                flipped[value] = [key]
            else:
                raise Exception('Non-unique prefix encountered')

    def __iter__(self):
        for source, prefix in self.sources.items():
            with utils.smart_open(source) as fin:
                for item_no, line in enumerate(fin):
                    yield LabeledSentence(utils.to_unicode(line).split(), [prefix + '_%s' % item_no])

    def to_array(self):
        self.sentences = []
        for source, prefix in self.sources.items():
            with utils.smart_open(source) as fin:
                for item_no, line in enumerate(fin):
                    self.sentences.append(LabeledSentence(utils.to_unicode(line).split(), [prefix + '_%s' % item_no]))
        return self.sentences

    def sentences_perm(self):
        shuffle(self.sentences)
        return self.sentences

sources = {'test.txt' : 'DOCS'}
sentences = LabeledLineSentence(sources)

model = Doc2Vec(min_count=1, window=10, size=100, sample=1e-4, negative=5, workers=8)
model.build_vocab(sentences.to_array())

for epoch in range(10):
    model.train(sentences.sentences_perm())

print(model.docvecs)
</code></pre>

<p>my test.txt file contains a paragraph per line.</p>

<p>The code runs fine and generates DocvecsArray for each line of text</p>

<p>my goal is to have an output like so:</p>

<p>cluster 1: [DOC_5,DOC_100,...DOC_N]<br>
cluster 2: [DOC_0,DOC_1,...DOC_N]</p>

<p>I have found the <a href="http://stackoverflow.com/questions/27889873/clustering-text-documents-using-scikit-learn-kmeans-in-python">following Answer</a>, but the output is:</p>

<p>cluster 1: [word,word...word]<br>
cluster 2: [word,word...word]</p>

<p>How can I alter the code and get document clusters?</p>

<p>So it looks like you're almost there.</p>

<p>You are outputting a set of vectors. For the sklearn package, you have to put those into a numpy array - using the numpy.toarray() function would probably be best. <a href="http://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_iris.html" rel="nofollow">The documentation</a> for KMeans is really stellar and even across the whole library it's good.</p>

<p>A note for you is that I have had much better luck with <a href="http://scikit-learn.org/stable/auto_examples/cluster/plot_dbscan.html#example-cluster-plot-dbscan-py" rel="nofollow">DBSCAN</a> than KMeans, which are both contained in the same sklearn library. DBSCAN doesn't require you to specify how many clusters you want to have on the output.</p>

<p>There are well-commented code examples in both links.</p>

python machine-learning k-means word2vec doc2vec