Calculating similarity "score" between multiple dictionaries
<p>I have a reference dictionary, "dictA" and I need to compare it (calculate similarity between key and vules) to n amount of dictionaries that are generated on the spot. Each dictionary has the same length. Lets say for the sake of the discussion that the n amount of dictionaries to compare it with is 3: dictB, dictC, dictD.</p>

<p>Here is how dictA looks like:</p>

<pre><code>dictA={'1':"U", '2':"D", '3':"D", '4':"U", '5':"U",'6':"U"}
</code></pre>

<p>Here are how dictB,dictC and dictD look like:</p>

<pre><code>dictB={'1':"U", '2':"U", '3':"D", '4':"D", '5':"U",'6':"D"}
dictC={'1':"U", '2':"U", '3':"U", '4':"D", '5':"U",'6':"D"}
dictD={'1':"D", '2':"U", '3':"U", '4':"U", '5':"D",'6':"D"}
</code></pre>

<p>I have a solution, but just for the option of two dictionaries:</p>

<pre><code>sharedValue = set(dictA.items()) &amp; set(dictD.items())
dictLength = len(dictA)
scoreOfSimilarity = len(sharedValue)
similarity = scoreOfSimilarity/dictLength
</code></pre>

<p>My questions is:
How can I iterate through n amount of dictionaries with dictA being a primary dictionary which I compare others with. The goal is to get a "similarity" value for each dictionary I am gonna iterate through against the primary dictionary. </p>

<p>Thanks for your help.</p>

<p>Based on your problem setup, there appears to be no alternative to looping through the input list of dictionaries. However, there is a multiprocessing trick that can be applied here. </p>

<p>Here is your input:</p>

<pre><code>dict_a = {'1': "U", '2': "D", '3': "D", '4': "U", '5': "U", '6': "U"}
dict_b = {'1': "U", '2': "U", '3': "D", '4': "D", '5': "U", '6': "D"}
dict_c = {'1': "U", '2': "U", '3': "U", '4': "D", '5': "U", '6': "D"}
dict_d = {'1': "D", '2': "U", '3': "U", '4': "U", '5': "D", '6': "D"}
other_dicts = [dict_b, dict_c, dict_d]
</code></pre>

<p>I have included @gary_fixler's map technique as <code>similarity1</code>, in addition to the <code>similarity2</code> function that I will use for the loop technique.</p>

<pre><code>def similarity1(a):
    def _(b):
        shared_value = set(a.items()) &amp; set(b.items())
        dict_length = len(a)
        score_of_similarity = len(shared_value)
        return score_of_similarity / dict_length
    return _

def similarity2(c):
    a, b = c
    shared_value = set(a.items()) &amp; set(b.items())
    dict_length = len(a)
    score_of_similarity = len(shared_value)
    return score_of_similarity / dict_length
</code></pre>

<p>We are evaluating 3 techniques here: <br />
(1) @gary_fixler's map <br />
(2) simple loop through the list of dicts<br />
(3) multiprocessing the list of dicts</p>

<p>Here are the execution statements:</p>

<pre><code>print(list(map(similarity1(dict_a), other_dicts)))
print([similarity2((dict_a, dict_v)) for dict_v in other_dicts])

max_processes = int(multiprocessing.cpu_count()/2-1)
pool = multiprocessing.Pool(processes=max_processes)
print([x for x in pool.map(similarity2, zip(itertools.repeat(dict_a), other_dicts))])
</code></pre>

<p>You will find that all 3 techniques produce the same result:</p>

<pre><code>[0.5, 0.3333333333333333, 0.16666666666666666]
[0.5, 0.3333333333333333, 0.16666666666666666]
[0.5, 0.3333333333333333, 0.16666666666666666]
</code></pre>

<p>Note that, for multiprocessing, you have <code>multiprocessing.cpu_count()/2</code> cores (with each core having hyper-threading). Assuming that you have nothing else running on your system, and your program has no I/O or synchronization needs (as is the case for our problem), you will often get optimum performance with <code>multiprocessing.cpu_count()/2-1</code> processes, the <code>-1</code> being for the parent process.</p>

<p>Now, to time the 3 techniques:</p>

<pre><code>print(timeit.timeit("list(map(similarity1(dict_a), other_dicts))",
                    setup="from __main__ import similarity1, dict_a, other_dicts", 
                    number=10000))

print(timeit.timeit("[similarity2((dict_a, dict_v)) for dict_v in other_dicts]",
                    setup="from __main__ import similarity2, dict_a, other_dicts", 
                    number=10000))

print(timeit.timeit("[x for x in pool.map(similarity2, zip(itertools.repeat(dict_a), other_dicts))]",
                    setup="from __main__ import similarity2, dict_a, other_dicts, pool", 
                    number=10000))
</code></pre>

<p>This produces the following results on my laptop:</p>

<pre><code>0.07092539698351175
0.06757041101809591
1.6528456939850003
</code></pre>

<p>You can see that the basic loop technique performs the best. The multiprocessing was significantly worse than the other 2 techniques, because of the overhead of creating processes and passing data back and forth. This does not mean that multiprocessing is not useful here. Quite the contrary. Look at the results for a larger number of input dictionaries:</p>

<pre><code>for _ in range(7):
    other_dicts.extend(other_dicts)
</code></pre>

<p>This extends the dictionary list to 384 items. Here are the timing results for this input:</p>

<pre><code>7.934810006991029
8.184540337068029
7.466550623998046
</code></pre>

<p>For any larger set of input dictionaries, the multiprocessing technique becomes the most optimum. </p>

python python-3.x