how to hash numpy arrays to check for duplicates
<p>I have searched around a bit for tutorials etc. to help with this problem but cant seem to find anything.   </p>

<p>I have two lists of n-dimensional numpy arrays (3D array form of some images) and am wanting to check for overlapping images within each list. Lets says list a is a training set and list b is a validation set.
One solution is just to use a nested loop and check if each pair of arrays is equal using <code>np.array(a[i], b[j])</code> but that is slow (each list has about 200,000 numpy arrays in it) and frankly quite disgusting.   </p>

<p>I was thinking a more elegant way of achieving this would be to hash each of the numpy arrays within each list and then compare each entry using these hash tables.  </p>

<p>Firstly, is this solution correct, and secondly, how would I go about achieving this?<br>
An example of some of the data is below.</p>

<pre><code>train_dataset[:3]
array([[[-0.5       , -0.49607843, -0.5       , ..., -0.5       ,
         -0.49215686, -0.5       ],
        [-0.49607843, -0.47647059, -0.5       , ..., -0.5       ,
         -0.47254902, -0.49607843],
        [-0.49607843, -0.49607843, -0.5       , ..., -0.5       ,
         -0.49607843, -0.49607843],
        ..., 
        [-0.49607843, -0.49215686, -0.5       , ..., -0.5       ,
         -0.49215686, -0.49607843],
        [-0.49607843, -0.47647059, -0.5       , ..., -0.5       ,
         -0.47254902, -0.49607843],
        [-0.5       , -0.49607843, -0.5       , ..., -0.5       ,
         -0.49607843, -0.5       ]],

       [[-0.5       , -0.5       , -0.5       , ...,  0.48823529,
          0.5       ,  0.1509804 ],
        [-0.5       , -0.5       , -0.5       , ...,  0.48431373,
          0.14705883, -0.32745099],
        [-0.5       , -0.5       , -0.5       , ..., -0.32745099,
         -0.5       , -0.49607843],
        ..., 
        [-0.5       , -0.44901961,  0.1509804 , ..., -0.5       ,
         -0.5       , -0.5       ],
        [-0.49607843, -0.49607843, -0.49215686, ..., -0.5       ,
         -0.5       , -0.5       ],
        [-0.5       , -0.49607843, -0.48823529, ..., -0.5       ,
         -0.5       , -0.5       ]],

       [[-0.5       , -0.5       , -0.5       , ..., -0.5       ,
         -0.5       , -0.5       ],
        [-0.5       , -0.5       , -0.5       , ..., -0.5       ,
         -0.5       , -0.5       ],
        [-0.5       , -0.5       , -0.49607843, ..., -0.5       ,
         -0.5       , -0.5       ],
        ..., 
        [-0.5       , -0.5       , -0.5       , ..., -0.48823529,
         -0.5       , -0.5       ],
        [-0.5       , -0.5       , -0.5       , ..., -0.5       ,
         -0.5       , -0.5       ],
        [-0.5       , -0.5       , -0.5       , ..., -0.5       ,
         -0.5       , -0.5       ]]], dtype=float32)
</code></pre>

<p>I appreciate the help in advance.</p>

<p>It's not so difficult to come up with <em>something</em>:</p>

<pre><code>from collections import defaultdict
import numpy as np

def arrayhash(arr):
    u = arr.view('u' + str(arr.itemsize))
    return np.bitwise_xor.reduce(u.ravel())

def do_the_thing(a, b, hashfunc=arrayhash):
    table = defaultdict(list)
    for i, a_i in enumerate(a):
        table[hashfunc(a_i)].append(i)

    indices = []
    for j, b_j in enumerate(b):
        candidates = table[hashfunc(b_j)]
        for i in candidates:
            if np.array_equiv(a[i], b_j):
                indices.append((i,j))

    return indices
</code></pre>

<p>But note:</p>

<ul>
<li><p>Checking for floating point equality is oftentimes a bad idea, because limited precision and rounding error. Famous example:</p>

<pre><code>&gt;&gt;&gt; 0.1 + 0.2 == 0.3
False
</code></pre></li>
<li><p>NaN's don't compare equal to themselves:</p>

<pre><code>&gt;&gt;&gt; np.nan == np.nan
False
</code></pre></li>
<li><p>The simple hash function above regards the bit-representation of floats, but this is problematic in the presence of negative-zero and signaling NaN's.</p></li>
</ul>

<p>See also the discussion in this question: <a href="http://stackoverflow.com/questions/650175/good-way-to-hash-a-float-vector">Good way to hash a float vector?</a></p>

python arrays numpy hash