Python matrix comparison
<p>I have big data like:</p>

<pre><code>{'a_1':0b110000,
 'a_2':0b001100,
 'a_3':0b000011,
 'b_1':0b100100,
 'b_2':0b000001,
 'c_1':0b100000,}
</code></pre>

<p>and so on... the structure of the data can be reorganized and is more to show, what I want to achieve. Rows of 'a' will never overlap by their sub-rows.
What would be a performant way, to get the best combinations of two (ab,ac) or three (abc) or more rows in terms of most matching values?
Hope, the questions is clear somehow, hard to describe :/
Maybe some matrix operations of numpy?</p>

<p>more info:
possible combinations of two would be ab,ac,bc. ab would check rows of a (a_1,a_2,a_3) against rows of b (b_1,b_2) each other. a_1 &amp; b_1 means 0b110000 &amp; 0b100100 and would give one result. a_1 &amp; b_2 means 0b110000 &amp; 0b000001 and would give no result. That would be the description of a solution by loops, but it is very slow, especially with combinations of 8 or so (not covered by example data).</p>

<p>maybe a more clear structure of the data:</p>

<pre><code>{'a': [0b110000,
       0b001100,
       0b000011],
 'b': [0b100100,
       0b000001],
 'c': [0b100000]}
</code></pre>

<p>Let me show, how I'm doing those calculations until now. The data-structure is kind of different, as I tried to start this question with an 'I thought' better structure...</p>

<pre><code>data = {'a':[1,1,2,2,3,3],
        'b':[4,5,5,5,4,5],
        'c':[6,7,7,7,6,7]}    

combine_count = 3
for config in combinations(['a','b','c'],combine_count):
    ret = {}
    for index,combined in enumerate(zip(*tuple(data.get(k) for k in config))):
        ret.setdefault(combined, []).append(index)

for k,v in ret.items():
    score = len(v)
    if score &gt;= 2:
        print(k,score)
</code></pre>

<p>my problem with this is, that especially the process of constructing combined with larger combine_count takes a lot of time.
data of course is a lot larger. It has about 231 keys with lists each of length ~60000. Also, the RAM consumption is too high. </p>

<p>Not sure about your triple evaluation* but you might be able to modify this to do what you want. I am assuming you will iterate through the combinations of a,b,c etc.</p>

<pre><code>#!/usr/bin/python
import numpy as np
import random
import time

A = [np.random.randint(0, 2**15, random.randint(1, 5)) + 2**16 for i in range(231)]
best_score = 0
tm = time.time()
for i, a in enumerate(A):
  for j, b in enumerate(A[1:]):
    for k, c in enumerate(A[2:]):
      an, bn, cn = len(a), len(b), len(c) #some shortcuts

      a_block = np.broadcast_to(a.reshape(an, 1, 1), (an, bn, cn))
      b_block = np.broadcast_to(b.reshape(1, bn, 1), (an, bn, cn))
      c_block = np.broadcast_to(c.reshape(1, 1, cn), (an, bn, cn))

      all_and = c_block &amp; b_block &amp; a_block

      all_score = ((all_and &amp; 1) + 
                   ((all_and &gt;&gt; 1) &amp; 1) +
                   ((all_and &gt;&gt; 2) &amp; 1) +
                   ((all_and &gt;&gt; 3) &amp; 1) +
                   ((all_and &gt;&gt; 4) &amp; 1) +
                   ((all_and &gt;&gt; 5) &amp; 1))
      ix = np.unravel_index(np.argmax(all_score), (an, bn, cn))
      if all_score[ix] &gt; best_score:
        print(i,j,k, ix, all_score[ix], a_block[ix], b_block[ix], c_block[ix])
        best_score = all_score[ix]
        best_abc = (i, j, k)
        best_ix = ix[:]

print(time.time() - tm)
print(best_score)
print(best_abc)
print(best_ix)
''' gives
0 0 0 (0, 2, 0) 2 95038 76894 78667
0 0 1 (0, 3, 1) 3 95038 70262 96242
0 0 2 (0, 2, 0) 4 95038 76894 96255
0 3 2 (0, 0, 0) 5 95038 96255 96255
4 3 2 (0, 0, 0) 6 96255 96255 96255
871.6093053817749
6
(4, 3, 2)
(0, 0, 0)
'''
</code></pre>

<p>EDIT * I think this code does: find the location (and value of) the maximum between a1&amp;b1&amp;c1, a2&amp;b1&amp;c1, a3&amp;b1&amp;c1, a1&amp;b2&amp;c1 etc which is possibly different from a1&amp;b1&amp;c1 | a2&amp;b1&amp;c1 | a3&amp;b1&amp;c1 | a1&amp;b2&amp;c1</p>

<p>EDIT2 More explicitly showing process of iterating over a pseudo dataset. a,b,c are arrays 1 to 5 long but numpy randint can't generate random numbers 60000 bits long, also I've not attempted to ensure all the numbers are unique (which would be quite easy to do) It takes about 15m on this not very powerful laptop, so that gives you a starting point for comparison.</p>

<p>A way to speed up the process might be to confine comparison to just two i.e. a,b to start with and keep a list of the high scorers then go through  each of those combinations &amp;ing against all the other entries in the list to select the highest scoring three way and.</p>

python performance numpy matrix comparison