How to not parse periods with sklearn TfidfVectorizer?
<p>I just picked up sklearn so pardon my blatant ignorance :)...Right now I am trying to figure out how TfidfVectorizer works and how to avoid splitting on periods.  </p>

<pre><code>from sklearn.feature_extraction.text import TfidfVectorizer

 docs= ("'CSC.labtrunk', 'CSC.datacenter', 'CSC.netbu', 'CSC.asr5k.general', 'CSC.ena', 'CSC.embu'", "'CSC.ena'", "'CSC.embu', 'CSC.security', 'CSC.ena'", "'CSC.embu', 'CSC.datacenter', 'CSC.labtrunk', 'CSC.content-security', 'CSC.ena', 'CSC.embu.dev', 'CSC.spv.custom-prods', 'CSC.voice', 'CSC.policy-mgmt', 'CSC.nuova'", "'CSC.embu', 'CSC.sys', 'CSC.policy-mgmt', 'CSC.content-security', 'CSC.datacenter'", "'CSC.asr5k.general'", "'CSC.sys'", "'CSC.labtrunk'")    
    vec = TfidfVectorizer()
    trfm_data = vec.fit_transform(docs)
    print trfm_data
</code></pre>

<p>Output sample:</p>

<pre><code>  (0, 6)    0.200552591995
  (0, 7)    0.200552591995
  (0, 8)    0.265074737928
  (0, 0)    0.265074737928
  (0, 11)   0.316288846342
  (0, 4)    0.228737749732
  (0, 9)    0.228737749732
  (0, 2)    0.757857197424


    print vec.inverse_transform(trfm_data)
</code></pre>

<p>Output sample:</p>

<pre><code>[u'embu', u'ena', u'general', u'asr5k', u'netbu', u'datacenter',
       u'labtrunk', u'csc']
</code></pre>

<p>Ideally, I'd like to treat each item as a string such as  <code>"'CSC.labtrunk', 'CSC.datacenter', 'CSC.netbu', 'CSC.asr5k.general', 'CSC.ena', 'CSC.embu'"</code>.</p>

<p>Not sure if It is proper convention, but I used list of strings rather than tuple of strings and got desired output.</p>

<p>sample data:</p>

<pre><code>data = ["'CSC.labtrunk', 'CSC.datacenter', 'CSC.netbu', 'CSC.asr5k.general', 'CSC.ena', 'CSC.embu'", "'CSC.ena'", "'CSC.embu', 'CSC.security', 'CSC.ena'", "'CSC.embu', 'CSC.datacenter', 'CSC.labtrunk', 'CSC.content-security', 'CSC.ena', 'CSC.embu.dev', 'CSC.spv.custom-prods', 'CSC.voice', 'CSC.policy-mgmt', 'CSC.nuova'", "'CSC.embu', 'CSC.sys', 'CSC.policy-mgmt', 'CSC.content-security', 'CSC.datacenter'", "'CSC.asr5k.general'", "'CSC.sys'", "'CSC.labtrunk'"]

vec = TfidfVectorizer(tokenizer=lambda i: i, lowercase=False)
trfm_data = vec.fit_transform(data)
data = trfm_data
trfm_data data
</code></pre>

<p>Sample output: </p>

<pre><code>[array(['CSC.embu', 'CSC.ena', 'CSC.asr5k.general', 'CSC.netbu',
       'CSC.datacenter', 'CSC.labtrunk'], 
      dtype='|S20'), array(['CSC.ena'], 
      dtype='|S20'), array(['CSC.security', 'CSC.embu', 'CSC.ena']
</code></pre>

python string scikit-learn vectorization