Clustering words on the basis of similarity in meaning which is provided in trainings set
<p>Which machine learning algorithm would serve to starting point for the below problem ?</p>

<p>I have a list of words which is similar to the below.</p>

<p><strong>curate</strong>  : a member of the clergy in certain churches (such as the Anglican church) who assists the priest in charge of a church or a group of churches</p>

<p><strong>yardman</strong>:  a person employed to do outdoor work (as mowing lawns)</p>

<p>and other normal words.</p>

<p>what i want to is decide that new word belongs to our category(job title) or not based on their meaning.</p>

<p>For example:</p>

<ul>
<li>nurse: 1</li>
<li>teacher:1</li>
<li>swimming:0</li>
<li>swimmer:1</li>
<li>facilitator:1</li>
<li>playing:0</li>
</ul>

<blockquote>
  <ul>
  <li>1-fit into our category</li>
  <li>2-does not fit into category.</li>
  </ul>
</blockquote>

<p>i am using python as tool for implementation machine learning purpose.</p>

<p>This is a problem of text similarity, where you want to match the similarity of a given block of text to a known set. For example, "nurse: a person working healthcare...". If the similarity is above a certain threshold, then there is a match, otherwise it is considered to not be a match.</p>

<p>To start, you will need some sort of representation for the text, so that you can perform the comparison. Some common text representations are:</p>

<ul>
<li>Bag of Words / Word counts (see <a href="http://scikit-learn.org/stable/modules/feature_extraction.html" rel="nofollow">TF-IDF</a>)</li>
<li>Word embeddings (see <a href="https://radimrehurek.com/gensim/models/word2vec.html" rel="nofollow">Word2Vec</a>)</li>
</ul>

<p>With TF-IDF, the sentences for each job description are converted into a vector of counts, with the length of the vector being equal to the size of the vocabulary for all words in all job descriptions. With Word2Vec, each word is treated as a vector of a defined size that represents the word's context in the training corpus. You can either trading your own Word2Vec model on your data or use a <a href="https://github.com/3Top/word2vec-api/blob/master/README.md" rel="nofollow">pretrained model</a>. You will need to combine the vectors for each sentence yourself to make the job descriptions comparable. An easy first step is to average all of the word vectors, but you can also try Doc2Vec (See <a href="https://radimrehurek.com/gensim/models/doc2vec.html" rel="nofollow">here</a>).</p>

<p>Lastly, you will need a distance function to compare the two representations. For TF-IDF, <a href="https://en.m.wikipedia.org/wiki/Cosine_similarity" rel="nofollow">cosine similarity</a> is a fairly standard one. For Word2Vec, you can use <a href="https://en.m.wikipedia.org/wiki/Euclidean_distance" rel="nofollow">euclidean distance</a> or even the word mover's distance (WMD see <a href="https://radimrehurek.com/gensim/models/doc2vec.html" rel="nofollow">here</a> again) </p>

<p>Putting it all together at a high level, you will want to do something like the following:</p>

<pre><code>trainingWords = {}
trainingWords['gardener'] = 'a person who tends a garden...'
trainingWords['valet'] = 'A person who parks a car...'

trainingVectors = {}

for (category in trainingWords):
    trainingVectors[category] = convert(trainingWords[category])

newWord = 'nurse'
newDescription = 'A person working in healthcare...'
newVector = convert(newDescription)

threshold = &lt;some number&gt;
output = {}
for (category in trainingVectors):
    if (distance(newVector, trainingVector[category]) &lt; threshold):
        output[category] = 1
    else:
        output[category] = 0

print output
</code></pre>

<p>This assumes that you have a function convert that converts raw text into one of the reorientation mentioned earlier, a function distance that users s distance function mentioned earlier, and a threshold appropriate to your problem and distance function. For example, cosine similarity uses values in the range of 0 to 1, so you'd need to pick a threshold for what is considered a match (ex. 0.85).</p>

python python-3.x machine-learning cluster-analysis word