Regex for unicode sentences without spaces using CountVectorizer
<p>I hope I don't have to provide an example set.</p>

<p>I have a 2D array where each array contains a set of words from sentences.</p>

<p>I am using a <a href="http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html" rel="nofollow">CountVectorizer</a> to effectively call <code>fit_transform</code> on the whole 2D array, such that I can build a vocabulary of words.</p>

<p>However, I have sentences like:</p>

<pre><code>u'Besides EU nations , Switzerland also made a high contribution at Rs 171 million LOCATION_SLOT~-nn+nations~-prep_besides nations~-prep_besides+made~prep_at made~prep_at+rs~num rs~num+NUMBER_SLOT'
</code></pre>

<p>And my current vectorizer is too strict at removing things like <code>~</code> and <code>+</code> as tokens. Whereas I want it to see each word in terms of <code>split()</code> a token in the vocab, i.e. <code>rs~num+NUMBER_SLOT</code> should be a word in itself in the vocab, as should <code>made</code>. At the same time, stopwords like <code>the</code> the <code>a</code> (the normal stopwords set) should be removed. </p>

<p>Current vectorizer: </p>

<pre><code>vectorizer = CountVectorizer(analyzer="word",stop_words=None,tokenizer=None,preprocessor=None,max_features=5000)
</code></pre>

<p>You can specify a <code>token_pattern</code> but I am not sure which one I could use to achieve my aims. Trying:</p>

<pre><code>token_pattern="[^\s]*"
</code></pre>

<p>Leads to a vocabulary of:</p>

<pre><code>{u'': 0, u'p~prep_to': 3764, u'de~dobj': 1107, u'wednesday': 4880, ...}
</code></pre>

<p>Which messes things up as <code>u''</code> is not something I want in my vocabulary.</p>

<p>What is the right token pattern for this type of <code>vocabulary_</code> I want to build?</p>

<p>I have figured this out. The vectorizer was allowing 0 or more non-whitespace items - it should allow 1 or more. The correct <code>CountVectorizer</code> is:</p>

<pre><code>CountVectorizer(analyzer="word",token_pattern="[\S]+",tokenizer=None,preprocessor=None,stop_words=None,max_features=5000)
</code></pre>

python nlp scikit-learn vectorization tokenize