Producing multi-level dictionary from word and part-of-speech
<p>Given some Penn Treebank tagged text in this format:</p>

<p>"David/NNP Short/NNP will/MD chair/VB the/DT meeting/NN ./. The/DT boy/NN sits/VBZ on/IN the/DT chair/NN ./."</p>

<p>I would like to produce a multi-level dictionary that has the word as a key and counts the frequency it appears tagged as each POS so we have ['Chair, VB : 1, NN : 1', 'The, DT : 3',] etc. </p>

<p>I figure I can use regexes to extract the word and the corresponding POS.</p>

<pre><code>r'[A+Za+z]+/' and r'/[A+Z]+'
</code></pre>

<p>But can't work out how to put this together to make an entry for a word and its corresponding POS occurences. </p>

<p>Thoughts?</p>

<p>You don't have to use regular expressions in this case.</p>

<p>What you can do is to split by space and then by slash collecting the results into a <a href="https://docs.python.org/2/library/collections.html#collections.defaultdict" rel="nofollow"><code>defaultdict</code></a> of <code>defaultdict</code> of <code>int</code>:</p>

<pre><code>In [1]: import re

In [2]: from collections import defaultdict

In [3]: s = "David/NNP Short/NNP will/MD chair/VB the/DT meeting/NN ./. The/DT boy/NN sits/VBZ on/IN the/DT chair/NN
   ...:  ./."

In [4]: d = defaultdict(lambda: defaultdict(int))

In [5]: for item in s.split():
   ...:     word, tag = item.split("/")
   ...:     word = word.lower()
   ...:     d[word][tag] += 1
</code></pre>

<p>Now the <code>d</code> would be:</p>

<pre><code>In [6]: for word, word_data in d.items():
    ...:     for tag, count in word_data.items():
    ...:         print(word, tag, count)
    ...:         
('boy', 'NN', 1)
('short', 'NNP', 1)
('on', 'IN', 1)
('david', 'NNP', 1)
('will', 'MD', 1)
('sits', 'VBZ', 1)
('chair', 'VB', 1)
('chair', 'NN', 1)
('.', '.', 2)
('meeting', 'NN', 1)
('the', 'DT', 3)
</code></pre>

python regex dictionary part-of-speech