sum of particular items of N lists using Python
<p>I have a task to analyse N texts using NLTK. Each text is longer than 100k words, so it is hard for computer to process so many data and that's why I decided to split each text after tokenizing in sublists like this:</p>

<p><code>chunks = [tokens_words[x:x+1000] for x in range (0,len(tokens_words), 1000)]</code></p>

<p>Probably, it works well.</p>

<p>And then I need to count, for example, number of nouns in each text. I do it like this:</p>

<pre><code>for chunk in chunks:
    for key in tagged.keys():
        for noun_tag in noun_tags:
            if tagged[key] == noun_tag:
                noun += 1
    totalNoun.append(noun)
</code></pre>

<p>then I use <code>sum()</code> and find percentage. I also tried <code>totalNoun += noun</code> but in both ways I receive smth like 3500% or 2498%. </p>

<p>What do I do wrong?</p>

<p>Let's say you have 2 chunks, one with 13 nouns and one with 30 nouns. Your code will do the following:</p>

<pre><code>noun: 0
totalNoun: []

# processing chunk 1 with 13 nouns...

noun : 13
totalNoun : [13]

# processing chunk 2 with 30 nouns...

noun : 43
totalNoun : [13,43]
</code></pre>

<p>As far as I can see, you do not set <code>noun</code> equal to 0 after each chunk. That is:</p>

<pre><code>for chunk in chunks:
    ## HERE ##
    noun = 0
    ## HERE ##

    for key in tagged.keys():
        for noun_tag in noun_tags:
            if tagged[key] == noun_tag:
                noun += 1
    totalNoun.append(noun)
</code></pre>

<p>Using <code>noun = 0</code> in your <code>for chunk in chunks</code> loop should do the trick.</p>

python python-2.7 nltk