Filtering out stopwords
<p>I've created a simple word count program and I'm trying to filter out commonly used words from my list using nltk (see below).</p>

<p>My question is how would I apply my "stop" filter to my "frequency" list?</p>

<pre><code>#Start 
from nltk.corpus import stopwords
import re
import string
frequency = {}
document_text = open('Import.txt', 'r')
text_string = document_text.read().lower()
match_pattern = re.findall(r'\b[a-z]{3,15}\b', text_string)

for word in match_pattern:
    count = frequency.get(word,0)
    frequency[word] = count + 1

frequency = {k:v for k,v in frequency.items() if v&gt;1}

stop = set(stopwords.words('english'))
stop = list(stop)
stop.append(".")

import csv

with open('Export.csv', 'w') as csvfile:
    writer = csv.writer(csvfile)
    for key, value in frequency.items():
       writer.writerow([key, value])
</code></pre>

<pre><code>stop = set(stopwords.words('english'))
stop.(".")

frequency = {k:v for k,v in frequency.items() if v&gt;1 and k not in stop}
</code></pre>

<p>While <code>stop</code> is still a <code>set</code>, check the keys of your <code>frequency</code> dictionary when doing the comprehension.  You can still make stop a list again afterwards.  </p>

<p>The reason I keep it as a set is because it is much more efficient to search sets than it is to search lists.</p>

python nltk stop-words