Create a DTM from large corpus
<p>I have a set of texts contained in a list, which I loaded from a csv file</p>

<p><code>texts=['this is text1', 'this would be text2', 'here we have text3']</code></p>

<p>and I would like to create a document-term matrix, by using <em>stemmed words</em>.
I have also stemmed them to have:</p>

<p><code>[['text1'], ['would', 'text2'], ['text3']]</code></p>

<p>What I would like to do is to create a DTM that counts all the stemmed terms (then  I would need to do some operations on the rows). </p>

<p>For what concerns the unstemmed texts, I am able to make the DTM for short texts, by using the function fn_tdm_df reported <a href="http://stackoverflow.com/questions/15899861/efficient-term-document-matrix-with-nltk">here</a>.
What would be more practical for me, though, is to make a DTM of the stemmed words. Just to be clearer, the output I have from applying "fn_tdm_df":</p>

<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-js lang-js prettyprint-override"><code>  be  have  here   is  text1  text2  text3  this   we  would
0  1.0   1.0   1.0  1.0    1.0    1.0    1.0     1  1.0    1.0
1  0.0   0.0   0.0  0.0    0.0    0.0    0.0     1  0.0    0.0</code></pre>
</div>
</div>
</p>

<p>First, I do not know why I have only two rows, instead of three. Second, my desired output would be something like:</p>

<pre><code>  text1  would  text2  text3
0   1      0      0      0
1   0      1      1      0
2   0      0      0      1
</code></pre>

<p>I am sorry but I am really desperate on this output. I also tried to export and reimport the stemmed texts on R, but it doesn't encode correctly. I would probably need to handle DataFrames, as for the huge amount of data. What would you suggest me?</p>

<p>----- UPDATE</p>

<p>Using CountVectorizer I am not fully satisfied, as I do not get a tractable matrix in which I can normalize and sum rows/columns easily.</p>

<p>Here is the code I am using, but it is blocking Python (dataset too large). How can I run it efficiently?</p>

<pre><code>vect = CountVectorizer(min_df=0., max_df=1.0)
X = vect.fit_transform(texts)
print(pd.DataFrame(X.A, columns=vect.get_feature_names()).to_string())
df = pd.DataFrame(X.toarray().transpose(), index = vect.get_feature_names())
</code></pre>

<p>Why don't you use <code>sklearn</code>? The <code>CountVectorizer()</code> method <strong>converts a collection of text documents to a matrix of token counts</strong>. What's more it gives a sparse representation of the counts using <code>scipy</code>.</p>

<p>You can either give your raw entries to the method or preprocess it as you have done (stemming + stop words).</p>

<p>Check this out : <a href="http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html" rel="nofollow">CountVectorizer()</a></p>

python pandas scikit-learn nltk