Text Pre-processing with NLTK
<p>I am practicing on using NLTK to remove certain features from raw tweets and subsequently hoping to remove tweets that are (to me) irelevant (e.g. empty tweet or single word tweets). However, it seems that some of the single word tweets are not removed. I am also facing an issue with not able to remove any stopword that are either at the beginning or end of sentence.</p>

<p>Any advice? At the moment, I hope to pass back a sentence as an output rather than a list of tokenized words.</p>

<p>Any other comment on improving the code (processing time, elegance) are welcome.</p>

<pre><code>import string
import numpy as np
import nltk
from nltk.corpus import stopwords

cache_english_stopwords=stopwords.words('english')
cache_en_tweet_stopwords=stopwords.words('english_tweet')

# For clarity, df is a pandas dataframe with a column['text'] together with other headers.

def tweet_clean(df):
    temp_df = df.copy()
    # Remove hyperlinks
    temp_df.loc[:, "text"] = temp_df.loc[:, "text"].replace('https?:\/\/.*\/\w*', '', regex=True)
    # Remove hashtags
    # temp_df.loc[:,"text"]=temp_df.loc[:,"text"].replace('#\w*', '', regex=True)
    temp_df.loc[:, "text"] = temp_df.loc[:, "text"].replace('#', ' ', regex=True)
    # Remove citations
    temp_df.loc[:, "text"] = temp_df.loc[:, "text"].replace('\@\w*', '', regex=True)
    # Remove tickers
    temp_df.loc[:, "text"] = temp_df.loc[:, "text"].replace('\$\w*', '', regex=True)
    # Remove punctuation
    temp_df.loc[:, "text"] = temp_df.loc[:, "text"].replace('[' + string.punctuation + ']+', '', regex=True)
    # Remove stopwords
    for tweet in temp_df.loc[:,"text"]:
        tweet_tokenized=nltk.word_tokenize(tweet)
        for w in tweet_tokenized:
            if (w.lower() in cache_english_stopwords) | (w.lower() in cache_en_tweet_stopwords):
                temp_df.loc[:, "text"] = temp_df.loc[:, "text"].replace('[\W*\s?\n?]'+w+'[\W*\s?]', ' ', regex=True)
                #print("w in stopword")
    # Remove quotes
    temp_df.loc[:, "text"] = temp_df.loc[:, "text"].replace('\&amp;*[amp]*\;|gt+', '', regex=True)
    # Remove RT
    temp_df.loc[:, "text"] = temp_df.loc[:, "text"].replace('\s+rt\s+', '', regex=True)
    # Remove linebreak, tab, return
    temp_df.loc[:, "text"] = temp_df.loc[:, "text"].replace('[\n\t\r]+', ' ', regex=True)
    # Remove via with blank
    temp_df.loc[:, "text"] = temp_df.loc[:, "text"].replace('via+\s', '', regex=True)
    # Remove multiple whitespace
    temp_df.loc[:, "text"] = temp_df.loc[:, "text"].replace('\s+\s+', ' ', regex=True)
    # Remove single word sentence
    for tweet_sw in temp_df.loc[:, "text"]:
        tweet_sw_tokenized = nltk.word_tokenize(tweet_sw)
        if len(tweet_sw_tokenized) &lt;= 1:
            temp_df.loc["text"] = np.nan
    # Remove empty rows
    temp_df.loc[(temp_df["text"] == '') | (temp_df['text'] == ' ')] = np.nan
    temp_df = temp_df.dropna()
    return temp_df
</code></pre>

<p>What is df? a list of tweets? 
You maybe should consider cleaning the tweet one after the other and not as a list of tweets. It would be easier to have a function <code>tweet_cleaner(single_tweet)</code>.</p>

<p>nltk provides a <a href="http://www.nltk.org/api/nltk.tokenize.html" rel="nofollow">TweetTokenizer</a> to clean the tweets.</p>

<p>the <a href="https://docs.python.org/3.5/library/re.html" rel="nofollow">"re" package</a> provides good solutions to use regex.</p>

<p>I advice you to create a variable for an easier use of <code>temp_df.loc[:, "text"]</code></p>

<p>Deleting stopwords in a sentence is described [here]  (<a href="http://stackoverflow.com/questions/19130512/stopword-removal-with-nltk#19133088">Stopword removal with NLTK</a>): 
<code>clean_wordlist = [i for i in sentence.lower().split() if i not in stopwords]</code></p>

<p>If you want to use regex (with the re package), you can</p>

<ol>
<li><p>create a regex pattern composed of all the stopwords (out of the tweet_clean function): 
<code>stop_pattern = re.compile('|'.join(stoplist)(?siu))</code><br>
(?siu) for multiline, ignorecase, unicode</p></li>
<li><p>and use this pattern to clean any string
<code>clean_string = stop_pattern.sub('', input_string)</code></p></li>
</ol>

<p>(you can concatenate the 2 stoplists if having separate ones is not needed)</p>

<p>To remove 1 words tweet you could only keep the one longest than 1 word:<br>
<code>if len(tweet_sw_tokenized) &gt;= 1:
    kept_ones.append(tweet_sw)</code></p>

python twitter nltk