What is regex for website domain to use in tokenizing while keeping punctuation apart from words?
<p>This is normal output: 
<a href="https://i.stack.imgur.com/fFfnQ.png" rel="nofollow"><img src="https://i.stack.imgur.com/fFfnQ.png" alt="enter image description here"></a></p>

<p>What I want is to keep domain names as single tokens. For ex: "<a href="https://www.twitter.com" rel="nofollow">https://www.twitter.com</a>" should remain as a single token. </p>

<p>My code:</p>

<pre><code>import nltk
from nltk.tokenize.regexp import RegexpTokenizer

line="My website: http://www.cartoon.com is not accessible."
pattern = r'^(((([A-Za-z0-9]+){1,63}\.)|(([A-Za-z0-9]+(\-)+[A-Za-z0-9]+){1,63}\.))+){1,255}$'
tokeniser=RegexpTokenizer(pattern)

print (tokeniser.tokenize(line))
</code></pre>

<p>Output:</p>

<pre><code>[]
</code></pre>

<p>What am I doing wrong? Any better regex for domain names?</p>

<p>Edit: The special character must remain as a separate token, like from above example, tokenization must separate ('website' , ':').</p>

<p>Use a 'standard' domain regex</p>

<pre><code>import re
line="My website: http://www.cartoon.com is not accessible."
print(re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&amp;+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', line))
</code></pre>

<p>Returns: ['<a href="http://www.cartoon.com" rel="nofollow">http://www.cartoon.com</a>']</p>

<p>Also works for like 99% of domain names</p>

python regex nltk tokenize