Getting the root word using the Wordnet Lemmatizer
<p>I need to find a common root word matched for all related words for a keyword extractor.</p>

<p>How to convert words into the same root using the python nltk lemmatizer? </p>

<ul>
<li>Eg: 

<ol>
<li>generalized, generalization -> general </li>
<li>optimal, optimized -> optimize (maybe) </li>
<li>configure, configuration, configured -> configure</li>
</ol></li>
</ul>

<p>The python nltk lemmatizer gives 'generalize', for 'generalized' and 'generalizing' when part of speech(pos) tag parameter is used but not for 'generalization'.</p>

<p>Is there a way to do this?</p>

<p>Use SnowballStemmer:</p>

<pre><code>&gt;&gt;&gt; from nltk.stem.snowball import SnowballStemmer
&gt;&gt;&gt; stemmer = SnowballStemmer("english")
&gt;&gt;&gt; print(stemmer.stem("generalized"))
general
&gt;&gt;&gt; print(stemmer.stem("generalization"))
general
</code></pre>

<blockquote>
  <p>Note: Lemmatisation is closely related to stemming. The difference is
  that a stemmer operates on a single word without knowledge of the
  context, and therefore cannot discriminate between words which have
  different meanings depending on part of speech.</p>
</blockquote>

<p>A general issue I have seen with lemmatizers is that it identifies even bigger words as <em>lemma</em>s.</p>

<p>Example: 
In WordNet Lemmatizer(checked in NLTK), </p>

<ul>
<li>Genralized => Generalize</li>
<li>Generalization => Generalization </li>
<li>Generalizations => Generalization</li>
</ul>

<p>POS tag was not given as input in the above cases, so it was always considered <em>noun</em>.</p>

python nlp nltk wordnet lemmatization