Python - NLTK separating punctuation
<p>i'm pretty new to Python, i'm trying to use NLTK to remove stopwords of my file.
The code is working, however it's separating punctuation, if my text is a tweet with a mention (@user), i get "@ user".
Later i'll need to do a word frequency, and i need mentions and hashtags to be working properly.
My code:</p>

<pre><code>from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
import codecs
arquivo = open('newfile.txt', encoding="utf8")
linha = arquivo.readline()
while linha:
    stop_word = set(stopwords.words("portuguese"))
    word_tokens = word_tokenize(linha)
    filtered_sentence = [w for w in word_tokens if not w in stop_word]
    filtered_sentence = []
    for w in word_tokens:
       if w not in stop_word:
           filtered_sentence.append(w)
    fp = codecs.open("stopwords.txt", "a", "utf-8")
    for words in (filtered_sentence):
        fp.write(words + " ")
    fp.write("\n")
    linha= arquivo.readline()
</code></pre>

<p><strong>EDIT</strong>
Not sure if this is the best way to do it, but i fixed it this way:</p>

<pre><code>for words in (filtered_sentence):
        fp.write(words)
        if words not in string.punctuation:
            fp.write(" ")
    fp.write("\n")
</code></pre>

<p>instead of <code>word_tokenize</code>, you could use <a href="http://www.nltk.org/api/nltk.tokenize.html#module-nltk.tokenize.casual" rel="nofollow">Twitter-aware tokenizer</a> provided by nltk:</p>

<pre><code>from nltk.tokenize import TweetTokenizer

...
tknzr = TweetTokenizer()
...
word_tokens = tknzr.tokenize(linha)
</code></pre>

python nltk