pairwise comparisons within a dataset
<p>My data is 18 vectors each with upto 200 numbers but some with 5 or other numbers.. organised as:</p>

<pre><code>[2, 3, 35, 63, 64, 298, 523, 624, 625, 626, 823, 824]
[2, 752, 753, 808, 843]
[2, 752, 753, 843]
[2, 752, 753, 808, 843]
[3, 36, 37, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, ...]
</code></pre>

<p>I would like to find the pair that is the most similar in this group of lists. The numbers themselves are not important, they may as well be strings - a 2 in one list and a 3 in another list are not comparable. </p>

<p>I am looking if the variables are the same. for example, the second list is exactly the same as the 4th list but only  1 variable different from list 3.</p>

<p>Additionally it would be nice to also find the most similar triplet or n that are the most similar, but pairwise is the first and most important task.</p>

<p>I hope i have layed out this problem clear enough but i am very happy to supply any more information that anyone might need!</p>

<p>I have a feeling it involves numpy or scipy norm/cosine calculations, but i cant quite work out how to do it, or if this is the best method.</p>

<p>Any help would be greatly appreciated!</p>

<p>You can use <code>itertools</code> to generate your pairwise comparisons. If you just want the items which are shared between two lists you can use a <code>set</code> intersection. Using your example:</p>

<pre class="lang-python prettyprint-override"><code>import itertools

a = [2, 3, 35, 63, 64, 298, 523, 624, 625, 626, 823, 824]
b = [2, 752, 753, 808, 843]
c = [2, 752, 753, 843]
d = [2, 752, 753, 808, 843]
e = [3, 36, 37, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112]

data = [a, b, c, d, e]

def number_same(a, b):
    # Find the items which are the same
    return set(a).intersection(set(b))

for i in itertools.permutations([i for i in range(len(data) - 1)], r=2):
    print "Indexes: ", i, len(number_same(data[i[0]], data[i[1]]))

&gt;&gt;&gt;Indexes  (0, 1) 1
Indexes  (0, 2) 1
Indexes  (0, 3) 1
Indexes  (1, 0) 1
Indexes  (1, 2) 4
Indexes  (1, 3) 5  ... etc 
</code></pre>

<p>This will give the number of items which are shared between two lists, you could maybe use this information to define which two lists are the best pair...</p>

python python-2.7 numpy scipy cosine-similarity