How to improve distance function in python
<p>I am trying to do a classification exercise on email docs (strings containing words). </p>

<p>I defined the distance function as following:</p>

<pre><code>def distance(wordset1, wordset2):

 if len(wordset1) &lt; len(wordset2):
    return len(wordset2) - len(wordset1)
 elif len(wordset1) &gt; len(wordset2):
    return len(wordset1) - len(wordset2)
 elif len(wordset1) == len(wordset2):
    return 0    
</code></pre>

<p>However, the accuracy in the end is pretty low (0.8). I guess this is because of the not so accurate distance function. How can I improve the function? Or what are other ways to calculate the "distance" between email docs?</p>

<p>One common measure of similarity for use in this situation is the <a href="https://nickgrattan.wordpress.com/2014/02/18/jaccard-similarity-index-for-measuring-document-similarity/" rel="nofollow">Jaccard similarity</a>. It ranges from 0 to 1, where 0 indicates complete dissimilarity and 1 means the two documents are identical. It is defined as </p>

<pre><code>wordSet1 = set(wordSet1)
wordSet2 = set(wordSet2)
sim = len(wordSet1.intersection(wordSet2))/len(wordSet1.union(wordSet2))
</code></pre>

<p>Essentially, it is the ratio of the intersection of the sets of words to the ratio of the union of the sets of words. This helps control for emails that are of different sizes while still giving a good measure of similarity. </p>

python distance knn