Converting DictVectorizer to TfIdfVectorizer
<p>I need to convert some data that I have in this format into a term document matrix: <a href="http://pastebin.com/u1A7v1CV" rel="nofollow">http://pastebin.com/u1A7v1CV</a></p>

<p>Essentially, each row represents a document represented as word_label_id and frequency. The words corresponding to each word_label_id are in a different file. </p>

<p>I want to convert this into a term document matrix so that I can vectorize the data and cluster it. </p>

<p>I have managed to convert the data to a dictionary and use DictVectorizer to get a one-hot encoded sparse representation because someone suggested I do this in the following way: </p>

<pre><code>data = []

with open('../data/input.mat', 'r') as file:
    for i, line in enumerate(file):
        l = line.split()
        d = dict([(k, v) for k, v in zip(l[::2], l[1::2])])
        data.append(d)


v = DictVectorizer(sparse=True, dtype=float)
X = v.fit_transform(data)
</code></pre>

<p>This is what the output looks like: </p>

<pre><code>  (0, 1312) 1.0
  (0, 2704) 1.0
  (0, 3322) 1.0
  (0, 3492) 1.0
  (0, 3506) 1.0
  (0, 3660) 1.0
  (0, 3674) 1.0
  (0, 3813) 1.0
  (0, 4782) 1.0
  (0, 4827) 1.0
  (0, 5208) 1.0
  (0, 5721) 1.0
  (0, 6105) 1.0
  (0, 6907) 1.0
  (0, 7252) 1.0
  (0, 7615) 1.0
  (0, 7890) 1.0
  (0, 7891) 1.0
  (0, 7906) 1.0
  (0, 7935) 1.0
  (0, 7954) 1.0
  (0, 7962) 1.0
  (0, 7986) 1.0
  (0, 8000) 1.0
  (0, 8012) 1.0
  : :
  (8579, 50731) 1.0
  (8579, 51298) 1.0
  (8579, 51686) 1.0
  (8579, 51732) 1.0
  (8579, 52439) 1.0
  (8579, 52563) 1.0
  (8579, 52621) 1.0
  (8579, 52980) 1.0
  (8579, 53013) 1.0
  (8579, 53018) 1.0
  (8579, 53155) 1.0
  (8579, 53180) 1.0
  (8579, 53317) 1.0
  (8579, 53739) 1.0
  (8579, 54114) 1.0
  (8579, 54444) 1.0
  (8579, 54489) 1.0
  (8579, 54922) 1.0
  (8579, 55074) 1.0
  (8579, 55164) 1.0
  (8579, 55311) 1.0
  (8579, 55741) 1.0
  (8579, 56010) 1.0
  (8579, 56062) 1.0
  (8579, 56946) 1.0
</code></pre>

<p>I don't know what this means and how to interpret it. Is this equivalent to the sparse matrix that you get out of <code>TfIdfVectorizer</code> in scikit-learn? </p>

<p>My next steps on this dataset are supposed to be feature selection and k-means clustering. I just don't know how to use the <code>DictVectorizer</code> or the <code>data</code> dictionary to proceed. </p>

<p>The output of the DictVectorizer is a SciPy sparse matrix just as you would have from TfIdfVectorizer. You can proceed with the feature selection and k-means clustering steps.</p>

python scikit-learn k-means tf-idf dictvectorizer