NLTK separately extract leaves and non-leaf nodes
<p>I'm working with the <a href="http://nlp.stanford.edu/sentiment/" rel="nofollow">Standford Sentiment Treebank</a> dataset and I'm attempting to extract the leaves and the nodes. The data is given follows </p>

<pre><code>(3 (2 (2 The) (2 Rock)) (4 (3 (2 is) (4 (2 destined) (2 (2 (2 (2 (2 to) (2 (2 be) (2 (2 the) (2 (2 21st) (2 (2 (2 Century) (2 's)) (2 (3 new) (2 (2 ``) (2 Conan)))))))) (2 '')) (2 and)) (3 (2 that) (3 (2 he) (3 (2 's) (3 (2 going) (3 (2 to) (4 (3 (2 make) (3 (3 (2 a) (3 splash)) (2 (2 even) (3 greater)))) (2 (2 than) (2 (2 (2 (2 (1 (2 Arnold) (2 Schwarzenegger)) (2 ,)) (2 (2 Jean-Claud) (2 (2 Van) (2 Damme)))) (2 or)) (2 (2 Steven) (2 Segal))))))))))))) (2 .)))
</code></pre>

<p>what I would like so for something as follows:</p>

<p>i) The leaves with the label (uni-gram):</p>

<pre><code>[(2 The), (2 Rock), (2 is), (2 destined),...]
</code></pre>

<p>ii) uper nodes with the labels (bi-gram):</p>

<pre><code>[(2 (2 the) (2 Rock)), (2 (2 ``) (2 Conan)), (2 (2 Century) (2 's)),..] 
</code></pre>

<p>until I get to the root of the tree. </p>

<p>I've attempted to use regex to accomplish this but it fails to output correctly.</p>

<p>The code I have (for the uni-gram):</p>

<pre><code>import re
import nltk

location = '.../NLP/Standford_Sentiment_Tree_Data_Set/' +\
           'trainDevTestTrees_PTB/trees/train.txt'
text = open(location, 'r')

test = text.readlines()[0]
text.close()

uni_regex = re.compile(r'(\([0-4] \w+\))')
temp01 = uni_regex.findall(test)

# bi-gram
bi_regex = re.compile(r'(\([0-4] \([0-4] \w+\) \([0-4] \w+\)\))')
temp02 = bi_regex.findall(test)
</code></pre>

<p>The above code outputs: </p>

<pre><code>['(2 The)', '(2 Rock)', '(2 is)', '(2 destined)', '(2 to)', '(2 be)', '(2 the)', '(2 21st)', '(2 Century)', '(3 new)',...]
</code></pre>

<p>and fails to capture <code>(2 ``)</code>, <code>(2 '')</code> and extracts <code>(2 Jean)</code> instead of <code>(2 Jean-Claude)</code></p>

<p>The output fails to capture <code>(2 (2``) (2 Conan))</code></p>

<p>Is there a way to get the result that I want using <code>nltk</code> or some configuration of <code>regex</code> that will not miss any tokens?  </p>

<p>I've had a look and attempted to modify the solution provided in <a href="http://stackoverflow.com/questions/25815002/nltk-tree-data-structure-finding-a-node-its-parent-or-children">NLTK tree data structure, finding a node, it&#39;s parent or children</a> but that question seems to deal with finding a specific word in a leave and the displaying the tree structure, whereas I require the indented solution to resemble the above n-grams.  </p>

<p>Don't  waste your time with regexps, this is what tree classes are for. Use the nltk's <code>Tree</code> class like this:</p>

<pre><code>mytree = "(3 (2 (2 The) (2 Rock)) (4 (3 (2 is) (4 (2 destined) (2 (2 (2 (2 (2 to) (2 (2 be) (2 (2 the) (2 (2 21st) (2 (2 (2 Century) (2 's)) (2 (3 new) (2 (2 ``) (2 Conan)))))))) (2 '')) (2 and)) (3 (2 that) (3 (2 he) (3 (2 's) (3 (2 going) (3 (2 to) (4 (3 (2 make) (3 (3 (2 a) (3 splash)) (2 (2 even) (3 greater)))) (2 (2 than) (2 (2 (2 (2 (1 (2 Arnold) (2 Schwarzenegger)) (2 ,)) (2 (2 Jean-Claud) (2 (2 Van) (2 Damme)))) (2 or)) (2 (2 Steven) (2 Segal))))))))))))) (2 .)))"

&gt;&gt;&gt; t = nltk.Tree.fromstring(mytree)
&gt;&gt;&gt; print(t)
(3
  (2 (2 The) (2 Rock))
  (4
    (3
      (2 is)
      (4
        (2 destined)
        (2
          ...
</code></pre>

<p>You can then extract and count the leaves, and request the corresponding "treepositions" (the path to each leaf, in the form of a list):</p>

<pre><code>&gt;&gt;&gt; leafpos = [ t.leaf_treeposition(n) for n, x in enumerate(t.leaves()) ]
&gt;&gt;&gt; print(leafpos[0:3])
[(0, 0, 0), (0, 1, 0), (1, 0, 0, 0)]
</code></pre>

<p>Finally, you can walk up the treepositions to get the units you want: the subtree dominated by the node immediately above each leaf, two steps above each leaf, etc:</p>

<pre><code>&gt;&gt;&gt; level1_subtrees = [ t[path[:-1]] for path in leafpos ]
&gt;&gt;&gt; for x in level1_subtrees:
...     print(x, end = " ")
(2 The) (2 Rock) (2 is) (2 destined) (2 to) (2 be) (2 the) ...

&gt;&gt;&gt; level2_subtrees = [ t[path[:-2]] for path in leafpos ]
</code></pre>

<p>Note, however, that higher-level subtrees don't look like you imagine. If you go up two levels from leaf 3 (<code>destined</code>), for example, you won't get a "bigram". You'll be at the node  labeled <code>4</code>, which dominates most of the rest of the sentence. Perhaps you're actually interested in enumerating all subtrees? In that case, just iterate over <code>t.subtrees()</code>. </p>

<p>If that's not what you want, take a look at the <code>Tree</code> API and pick out another way to select the parts you need.</p>

python tree nlp nltk nodes