nltk semantic word substitution
<p>I'm trying to find different ways of writing "events in [city]" which are semantically similar. I am trying to do this by finding words that are semantically similar to "events" so I can substitute them in. </p>

<p>To find these words I'm using nltk's wordnet corpus, but I'm getting some pretty strange results. For example, using the hyponyms of 'event.n.01', I'm getting "Miracles in Ottawa". </p>

<p>co-hyponyms and hypernyms seem just as bad or worse. I wonder if anyone understands the structure better and can offer a potential solution?</p>

<p>Here's some sample code:</p>

<pre><code>!/usr/bin/python3

import nltk

lemma = 'event.n.01'
synset = nltk.corpus.wordnet.synset(lemma)

print("%s: %s" % (synset.name(), synset.definition()))

print("\nFinding hyponyms...")
print([s.split('.')[0] for w in synset.hyponyms() for s in w.lemma_names()])

print("\nFinding hypernym paths...")
print([s.split('.')[0] for hyprs in synset.hypernym_paths() for hypr in hyprs for s in hypr.lemma_names()])

print("\nFinding co-hyponyms...")
for hypers in synset.hypernym_paths():
        for hyper in hypers:
                print(hyper.name())
                for hypos in hyper.hyponyms():
                        print("\t%s" % (', '.join(hypos.lemma_names())))

print(synset.similar())
</code></pre>

<p>The <em>hyponyms</em> of "event" are types of "event". One of them is "miracle", some others are:</p>

<pre><code>&gt;&gt;&gt; [s for w in synset.hyponyms() for s in w.lemma_names][:7]  # is 7 enough? :)
['zap', 'act', 'deed', 'human_action', 'human_activity', 'happening', 'occurrence']
</code></pre>

<p>"Event's" <em>hypernyms</em> are the oposite. Terms that "event" is a type of:</p>

<pre><code>&gt;&gt;&gt; synset.hypernyms()
[Synset('psychological_feature.n.01')]
</code></pre>

<p>You can see that "event" is one of it's <em>hyponyms</em>:</p>

<pre><code>&gt;&gt;&gt; synset.hypernyms()[0].hyponyms()
[Synset('motivation.n.01'), Synset('cognition.n.01'), Synset('event.n.01')]
</code></pre>

<p>Those are not really "similar" terms ("Psychological features in Ottawa" may seem like a correct result to a robot, but not to humans).</p>

<p>Perhaps it is better to go at it from a completely different angle, e.g.</p>

<pre><code>&gt;&gt;&gt; text = nltk.Text(word.lower() for word in nltk.corpus.brown.words())
&gt;&gt;&gt; text.similar('event')
time day man order state way case house one place action night point
situation work year act and area audience
</code></pre>

<p>Now, take those and sort them e.g. by path_similarity:</p>

<pre><code>&gt;&gt;&gt; words = 'time day man order state way case house one place action night point'\
...         ' situation work year act and area audience'.split()
&gt;&gt;&gt; 
&gt;&gt;&gt; def get_symilarity(synset, word):
...     return max([synset.path_similarity(synset2)
...                for synset2 in nltk.corpus.wordnet.synsets(word)]+[0])
&gt;&gt;&gt; 
&gt;&gt;&gt; sorted(words, key=lambda w: get_symilarity(synset, w), reverse=True)[:5]
['act', 'case', 'action', 'time', 'way']
</code></pre>

<p>Is that a good result? I don't know. I guess it could work: "Acts in Ottawa", "Cases in New York", "Action in Rome", "Time in Tokyo", "Ways in Amsterdam"...</p>

python nlp nltk