How to faster compute the count frequency of words in a large words list with python and be a dictionary
<p>There is a very long words list, the length of list is about 360000. I want to get the each word frequency, and to be a dictionary. </p>

<p>For example:</p>

<pre><code>{'I': 50, 'good': 30,.......}
</code></pre>

<p>Since the word list is large,  I found it take a lot of time to compute it. Do you have faster method to accomplish this?</p>

<p>My code, so far, is the following:</p>

<pre><code>  dict_pronoun = dict([(i, lst_all_tweet_noun.count(i)) for i in 
                        lst_all_tweet_noun])
  sorted(dict_pronoun)
</code></pre>

<p>You are doing several things wrong here:</p>

<ul>
<li><p>You are building a huge list first, then turn that list object into a dictionary. There is no need to use the <code>[..]</code> list comprehension; just dropping the <code>[</code> and <code>]</code> would turn it into a much more memory-efficient generator expression.</p></li>
<li><p>You are using <code>dict()</code> with a loop instead of a <code>{keyexpr: valueexpr for ... in ...}</code> dictionary comprehension; this would avoid a generator expression altogether and go straight to building a dictionary.</p></li>
<li><p>You are using <code>list.count()</code>, this does a <strong>full scan</strong> of the list for every element. You turned a linear scan to count N items into a O(N**2) quadratic problem. You could simply increment an integer in the dictionary each time you find the key already is present, set the value to 0 otherwise, but there are better options (see below).</p></li>
<li><p>The <code>sorted()</code> call is busy-work; it returns a sorted list of keys that is then discarded again. Dictionaries are not sortable, not and produce a dictionary again at any rate.</p></li>
</ul>

<p>Use a <a href="https://docs.python.org/3/library/collections.html#collections.Counter" rel="nofollow"><code>collections.Counter()</code> object</a> here to do your counting; it uses a linear scan:</p>

<pre><code>from collections import Counter

dict_pronoun = Counter(lst_all_tweet_noun)
</code></pre>

<p>A <code>Counter</code> has a <a href="https://docs.python.org/3/library/collections.html#collections.Counter.most_common" rel="nofollow"><code>Counter.most_common()</code> method</a> which will efficiently give you output <em>sorted by counts</em>, which is what I suspect you wanted to achieve with the <code>sorted()</code> call.</p>

<p>For example, to get the top K elements (where K is smaller than N, the size of the dictionary), a <code>heapq</code> is used to get you those elements in O(NlogK) time (avoiding a full O(NlogN) sort).</p>

python performance list python-3.x dictionary