How to update nltk package so that it does not break email into 3 different tokens?
<p>When I type following code:
<code>tokens = word_tokenize("a@b.com")</code></p>

<p>It gets broken into these 3 tokens: 'a' , '@' , 'b.com'</p>

<p>What I want to do, is to keep it as a single token 'a@b.com'.</p>

<p><em>DISCLAIMER: There are a lot of email regexps out there. I am not trying to match all email formats in this question, just showing an example</em>.</p>

<p>A regex approach with <code>RegexpTokenizer</code> (<a href="http://stackoverflow.com/questions/39777806/how-to-update-nltk-package-so-that-it-does-not-break-email-into-3-different-toke#comment66852179_39777806">mentioned above</a> by <a href="http://stackoverflow.com/users/1698431/lenz">lenz</a>) can work:</p>

<pre><code>from nltk.tokenize.regexp import RegexpTokenizer
line="My email: a@bc.com is not accessible."
pattern = r'\S+@[^\s.]+\.[a-zA-Z]+|\w+|[^\w\s]'
tokeniser=RegexpTokenizer(pattern)
tokeniser.tokenize(line)
# =&gt; ['My', 'email', ':', 'a@bc.com', 'is', 'not', 'accessible', '.']
</code></pre>

<p>The regex matches:</p>

<ul>
<li><code>\S+@[^\s.]+\.[a-zA-Z]+</code> - text looking like email:

<ul>
<li><code>\S+</code>  - 1 or more non-whitespace chars</li>
<li><code>@</code> - a <code>@</code> symbol</li>
<li><code>[^\s.]+</code> - 1 or more chars other than whitespaces and <code>.</code></li>
<li><code>\.</code> - a literal dot</li>
<li><code>[a-zA-Z]+</code> - 1 or more ASCII letters</li>
</ul></li>
<li><code>|</code> - or </li>
<li><code>\w+</code> - 1 or more word chars (letters, digits, or underscores)</li>
<li><code>|</code> - or</li>
<li><code>[^\w\s]</code> - a single (add <code>+</code> after it to match a sequence of 1 or more) occurrence of a char other than a word and whitespace char.</li>
</ul>

<p>See the <a href="https://regex101.com/r/bDfeYV/1" rel="nofollow">online regex demo</a>.</p>

python regex nlp nltk