How to split a string on commas or periods in nltk
<p>I want to separate a string on commas and/or periods in nltk. I've tried with <code>sent_tokenize()</code> but it separates only on periods.</p>

<p>I've also tried this code</p>

<pre><code>from nltk.tokenize.punkt import PunktSentenceTokenizer, PunktLanguageVars
ex_sent = "This is an example showing sentence filtration.This is how it is done, in case of Python I want to learn more. So, that i can have some experience over it, by it I mean python."
class CommaPoint(PunktLanguageVars):
    sent_end_chars = ('.','?','!',',')
tokenizer = PunktSentenceTokenizer(lang_vars = CommaPoint())
n_w=tokenizer.tokenize(ex_sent)
print n_w
</code></pre>

<p>The output for the code above is</p>

<pre><code>['This is an example showing sentence filtration.This is how it is done,' 'in case of Python I want to learn more.' 'So,' 'that i can have some experience over it,' 'by it I mean python.\n']
</code></pre>

<p>When I try to give '.' without any space it is taking it as a word </p>

<p>I want the output as</p>

<pre><code>['This is an example showing sentence filtration.' 'This is how it is done,' 'in case of Python I want to learn more.' 'So,' 'that i can have some experience over it,' 'by it I mean python.']
</code></pre>

<p><a href="http://i.stack.imgur.com/4pwEO.png" rel="nofollow">image of the above written code</a></p>

<p>How about something simpler with <code>re</code>:</p>

<pre><code>&gt;&gt;&gt; import re
&gt;&gt;&gt; sent = "This is an example showing sentence filtration.This is how it is done, in case of Python I want to learn more. So, that i can have some experience over it, by it I mean python."
&gt;&gt;&gt; re.split(r'[.,]', sent)
['This is an example showing sentence filtration', 'This is how it is done', ' in case of Python I want to learn more', ' So', ' that i can have some experience over it', ' by it I mean python', '']
</code></pre>

<p>To keep the delimiter, you can use group:</p>

<pre><code>&gt;&gt;&gt; re.split(r'([.,])', sent)
['This is an example showing sentence filtration', '.', 'This is how it is done', ',', ' in case of Python I want to learn more', '.', ' So', ',', ' that i can have some experience over it', ',', ' by it I mean python', '.', '']
</code></pre>

python nltk