scikit-learn vectorizing with big dataset
<p>I have 9GB of segmented documents on my disk and my vps only has 4GB memory.</p>

<p>How can I vectorize all the data set without loading all the corpus at initialization? Is there any sample code? </p>

<p>my code is as follows:</p>

<pre><code>contents = [open('./seg_corpus/' + filename).read()
            for filename in filenames]
vectorizer = CountVectorizer(stop_words=stop_words)
vectorizer.fit(contents)
</code></pre>

<p>Try this, instead of loading all texts into memory you can pass only handles to files into <code>fit</code> method, but you must specify <code>input='file'</code> in <code>CountVectorizer</code> constructor.</p>

<pre><code>contents = [open('./seg_corpus/' + filename)
        for filename in filenames]
vectorizer = CountVectorizer(stop_words=stop_words, input='file')
vectorizer.fit(contents)
</code></pre>

python numpy machine-learning scikit-learn