NLTK BigramTagger does not tag half of the sentence
<p>Can someone please explain the behaviour of NLTK's BigramTagger in these examples?</p>

<p>I instantiated the tagger by </p>

<pre><code>bi= BigramTagger(brown.tagged_sents(categories='news')[:500])
</code></pre>

<p>Now, I want to use this on one specific sentence.</p>

<pre><code>&gt;&gt;&gt; bi.tag(brown_sents[2])
[(u'The', u'AT'), (u'September-October', u'NP'), (u'term', u'NN'), (u'jury', u'NN'), (u'had', u'HVD'), (u'been', u'BEN'), (u'charged', u'VBN'), (u'by', u'IN'), (u'Fulton', u'NP-TL'), (u'Superior', u'JJ-TL'), (u'Court', u'NN-TL'), (u'Judge', u'NN-TL'), (u'Durwood', u'NP'), (u'Pye', u'NP'), (u'to', u'TO'), (u'investigate', u'VB'), (u'reports', u'NNS'), (u'of', u'IN'), (u'possible', u'JJ'), (u'``', u'``'), (u'irregularities', u'NNS'), (u"''", u"''"), (u'in', u'IN'), (u'the', u'AT'), (u'hard-fought', u'JJ'), (u'primary', u'NN'), (u'which', u'WDT'), (u'was', u'BEDZ'), (u'won', u'VBN'), (u'by', u'IN'), (u'Mayor-nominate', u'NN-TL'), (u'Ivan', u'NP'), (u'Allen', u'NP'), (u'Jr.', u'NP'), (u'.', u'.')]
</code></pre>

<p>Works well, but hey, it's all known data.
Let me change one word and see if it sets something off.</p>

<pre><code>&gt;&gt;&gt; sent=brown_sents[2]
&gt;&gt;&gt; sent[5]
u'been'
&gt;&gt;&gt; sent[5] = u'was'
&gt;&gt;&gt; bi.tag(sent)
[(u'The', u'AT'), (u'September-October', u'NP'), (u'term', u'NN'), (u'jury', u'NN'), (u'had', u'HVD'), (u'was', None), (u'charged', None), (u'by', None), (u'Fulton', None), (u'Superior', None), (u'Court', None), (u'Judge', None), (u'Durwood', None), (u'Pye', None), (u'to', None), (u'investigate', None), (u'reports', None), (u'of', None), (u'possible', None), (u'``', None), (u'irregularities', None), (u"''", None), (u'in', None), (u'the', None), (u'hard-fought', None), (u'primary', None), (u'which', None), (u'was', None), (u'won', None), (u'by', None), (u'Mayor-nominate', None), (u'Ivan', None), (u'Allen', None), (u'Jr.', None), (u'.', None)]
</code></pre>

<p>Now I expected to see changed tuple, <code>(u'been', u'BEN')</code> to now be (u'been', None). Why is everything after it in the sentence now not tagged? Those words were tagged in connection to another ones, not 'been'.</p>

<p>Any recommendation on use of tagged sentences would be appreciated as well.</p>

<p>You have to set a backoff tagger when using *gramTagger so that if the specific ngram is not seen in the training data, it will backoff to a tagger trained on a lower order ngram. See "Combining Taggers" section in <a href="http://www.nltk.org/book/ch05.html" rel="nofollow">http://www.nltk.org/book/ch05.html</a></p>

<pre><code>&gt;&gt;&gt; from nltk import DefaultTagger, UnigramTagger, BigramTagger
&gt;&gt;&gt; from nltk.corpus import brown
&gt;&gt;&gt; text = brown.tagged_sents(categories='news')[:500]
&gt;&gt;&gt; t0 = DefaultTagger('NN')
&gt;&gt;&gt; t1 = UnigramTagger(text, backoff=t0)
&gt;&gt;&gt; t2 = BigramTagger(text, backoff=t1)

&gt;&gt;&gt; test_sent = brown.sents()[502]
&gt;&gt;&gt; test_sent
[u'Noting', u'that', u'Plainfield', u'last', u'year', u'had', u'lost', u'the', u'Mack', u'Truck', u'Co.', u'plant', u',', u'he', u'said', u'industry', u'will', u'not', u'come', u'into', u'this', u'state', u'until', u'there', u'is', u'tax', u'reform', u'.']
&gt;&gt;&gt; t2.tag(test_sent)
[(u'Noting', u'VBG'), (u'that', u'CS'), (u'Plainfield', u'NP-HL'), (u'last', u'AP'), (u'year', u'NN'), (u'had', u'HVD'), (u'lost', u'VBD'), (u'the', u'AT'), (u'Mack', 'NN'), (u'Truck', 'NN'), (u'Co.', u'NN-TL'), (u'plant', 'NN'), (u',', u','), (u'he', u'PPS'), (u'said', u'VBD'), (u'industry', 'NN'), (u'will', u'MD'), (u'not', u'*'), (u'come', u'VB'), (u'into', u'IN'), (u'this', u'DT'), (u'state', u'NN'), (u'until', 'NN'), (u'there', u'EX'), (u'is', u'BEZ'), (u'tax', 'NN'), (u'reform', 'NN'), (u'.', u'.')]
</code></pre>

<p>And to show that it works with your example in the question ;P</p>

<pre><code>&gt;&gt;&gt; test_sent = brown.sents()[2]
&gt;&gt;&gt; test_sent
[u'The', u'September-October', u'term', u'jury', u'had', u'been', u'charged', u'by', u'Fulton', u'Superior', u'Court', u'Judge', u'Durwood', u'Pye', u'to', u'investigate', u'reports', u'of', u'possible', u'``', u'irregularities', u"''", u'in', u'the', u'hard-fought', u'primary', u'which', u'was', u'won', u'by', u'Mayor-nominate', u'Ivan', u'Allen', u'Jr.', u'.']
&gt;&gt;&gt; t2.tag(test_sent)
[(u'The', u'AT'), (u'September-October', u'NP'), (u'term', 'NN'), (u'jury', u'NN'), (u'had', u'HVD'), (u'been', u'BEN'), (u'charged', u'VBN'), (u'by', u'IN'), (u'Fulton', u'NP-TL'), (u'Superior', u'JJ-TL'), (u'Court', u'NN-TL'), (u'Judge', u'NN-TL'), (u'Durwood', u'NP'), (u'Pye', u'NP'), (u'to', u'TO'), (u'investigate', u'VB'), (u'reports', u'NNS'), (u'of', u'IN'), (u'possible', u'JJ'), (u'``', u'``'), (u'irregularities', u'NNS'), (u"''", u"''"), (u'in', u'IN'), (u'the', u'AT'), (u'hard-fought', u'JJ'), (u'primary', 'NN'), (u'which', u'WDT'), (u'was', u'BEDZ'), (u'won', u'VBN'), (u'by', u'IN'), (u'Mayor-nominate', u'NN-TL'), (u'Ivan', u'NP'), (u'Allen', u'NP'), (u'Jr.', u'NP'), (u'.', u'.')]
</code></pre>

<p>At some point, you might realize that <a href="http://stackoverflow.com/questions/30821188/python-nltk-pos-tag-not-returning-the-correct-part-of-speech-tag">Python NLTK pos_tag not returning the correct part-of-speech tag</a> </p>

python nlp nltk n-gram pos-tagger