What is regex for website domain to use in tokenizing while keeping punctuation apart from words?
<p>This is normal output: 
<a href="https://i.stack.imgur.com/fFfnQ.png" rel="nofollow"><img src="https://i.stack.imgur.com/fFfnQ.png" alt="enter image description here"></a></p>

<p>What I want is to keep domain names as single tokens. For ex: "<a href="https://www.twitter.com" rel="nofollow">https://www.twitter.com</a>" should remain as a single token. </p>

<p>My code:</p>

<pre><code>import nltk
from nltk.tokenize.regexp import RegexpTokenizer

line="My website: http://www.cartoon.com is not accessible."
pattern = r'^(((([A-Za-z0-9]+){1,63}\.)|(([A-Za-z0-9]+(\-)+[A-Za-z0-9]+){1,63}\.))+){1,255}$'
tokeniser=RegexpTokenizer(pattern)

print (tokeniser.tokenize(line))
</code></pre>

<p>Output:</p>

<pre><code>[]
</code></pre>

<p>What am I doing wrong? Any better regex for domain names?</p>

<p>Edit: The special character must remain as a separate token, like from above example, tokenization must separate ('website' , ':').</p>

<p>You may use</p>

<pre><code>tokeniser=RegexpTokenizer(r'\b(?:http|ftp)s?://\S*\w|\w+|[^\w\s]+')
</code></pre>

<p>See the <a href="https://regex101.com/r/HazY6N/1" rel="nofollow">regex demo</a></p>

<p><strong>Details</strong>:</p>

<ul>
<li><code>\b</code> - leading word boundary (there must be a non-word char before...)</li>
<li><code>(?:http|ftp)s?://</code> - a protocol, <code>http</code>/<code>https</code>, <code>ftp</code>/<code>ftps</code></li>
<li><code>\S*</code> - 0+ non-whitespace symbols</li>
<li><code>\w</code> - a word char (=letter/digit/<code>_</code>)</li>
<li><code>|</code> - or</li>
<li><code>\w+</code> - 1 or more word chars</li>
<li><code>|</code> - or </li>
<li><code>[^\w\s]+</code> - 1 or more non-word chars excluding whitespaces.</li>
</ul>

python regex nltk tokenize