How to preprocess twitter text data using python
<p>I have text data after retrieval from a <code>mongoDB</code> in this format: </p>

<p>**</p>

<pre><code>[u'In', u'love', u'#Paralympics?\U0001f60d', u"We've", u'got', u'nine', u'different', u'sports', u'live', u'streams', u'https://not_a_real_link', u't_https://anotherLink']

[u't_https://somelink']

[u'RT', u'@sportvibz:', u'African', u'medal', u'table', u'#Paralympics', u't_https://somelink', u't_https://someLink']
</code></pre>

<p>**</p>

<p>However I would like to replace all URLs in the list with the word 'URL' while preserving other texts in the list, i.e to something like this:</p>

<pre><code>[u'In', u'love', u'#Paralympics?\U0001f60d', u"We've", u'got', u'nine', u'different', u'sports', u'live', u'streams', u'URL', u'URL']
</code></pre>

<p>But when I run the code for stopword removal and also perform regular expression I get this result sample : </p>

<p>**</p>

<pre><code>In

URL

RT
</code></pre>

<p>**</p>

<p>Please could anyone help with this, as I'm finding this difficult.</p>

<p>Here is the code I have at the moment: </p>

<pre><code>def stopwordsRemover(self, rawText):
    stop = stopwords.words('english')
    ##remove stop words from the rawText argument and store the result list in processedText variable
    processedText = [i for i in rawText.split() if i not in stop]
    return processedText


def clean_text(self, rawText):
    temp_raw = rawText
    for i, text in enumerate(temp_raw):
        temp = re.sub(r'https?:\/\/.*\/[a-zA-Z0-9]*', 'URL', text)
    return temp
</code></pre>

<p>This is wrong:</p>

<pre><code>def clean_text(self, rawText):
    temp_raw = rawText
    for i, text in enumerate(temp_raw):
        temp = re.sub(r'https?:\/\/.*\/[a-zA-Z0-9]*', 'URL', text)
    return temp
</code></pre>

<p>you return the last substituted string instead of a list, that should replace your <code>rawText</code> input list (I must admit I'm puzzled by the fast that you seem to get the <em>first</em> item, but I'm still confident on the explanation)</p>

<p>do that instead:</p>

<pre><code>def clean_text(self, rawText):
    temp = list()
    for text in rawText:
        temp.append(re.sub(r'https?:\/\/.*\/\w*', 'URL', text))  # simpler regex with \w
    return temp
</code></pre>

<p>with a listcomp:</p>

<pre><code>def clean_text(self, rawText):
   return [re.sub(r'https?:\/\/.*\/\w*', 'URL', text) for text in rawText]
</code></pre>

<p>you could also work in-place, modifying <code>rawText</code> directly:</p>

<pre><code>def clean_text(self, rawText):
    rawText[:] = [re.sub(r'https?:\/\/.*\/\w*', 'URL', text) for text in rawText]
</code></pre>

python regex