What data structure should I use to process large amounts of text data?
<p>I'm trying to do some text classication using scikit-learn's TfidfVectorizer and the Nearest Neighbor algorithm.</p>

<p>I need to find similarity metrics between two data sets containing 18000 entries each. I'm unsure about what data structure can be best used to calculate what I think should be 18000*18000 similarity metrics.</p>

<p>I've only considered DataFrames so far.</p>

<p>If you don't need any intermediate data for further analyses you can use a generator to hold your data points then run the algorithms through generators calls. Otherwise you'd probably want a list.</p>

python numpy scikit-learn text-processing