how to hash numpy arrays to check for duplicates
<p>I have searched around a bit for tutorials etc. to help with this problem but cant seem to find anything.   </p>

<p>I have two lists of n-dimensional numpy arrays (3D array form of some images) and am wanting to check for overlapping images within each list. Lets says list a is a training set and list b is a validation set.
One solution is just to use a nested loop and check if each pair of arrays is equal using <code>np.array(a[i], b[j])</code> but that is slow (each list has about 200,000 numpy arrays in it) and frankly quite disgusting.   </p>

<p>I was thinking a more elegant way of achieving this would be to hash each of the numpy arrays within each list and then compare each entry using these hash tables.  </p>

<p>Firstly, is this solution correct, and secondly, how would I go about achieving this?<br>
An example of some of the data is below.</p>

<pre><code>train_dataset[:3]
array([[[-0.5       , -0.49607843, -0.5       , ..., -0.5       ,
         -0.49215686, -0.5       ],
        [-0.49607843, -0.47647059, -0.5       , ..., -0.5       ,
         -0.47254902, -0.49607843],
        [-0.49607843, -0.49607843, -0.5       , ..., -0.5       ,
         -0.49607843, -0.49607843],
        ..., 
        [-0.49607843, -0.49215686, -0.5       , ..., -0.5       ,
         -0.49215686, -0.49607843],
        [-0.49607843, -0.47647059, -0.5       , ..., -0.5       ,
         -0.47254902, -0.49607843],
        [-0.5       , -0.49607843, -0.5       , ..., -0.5       ,
         -0.49607843, -0.5       ]],

       [[-0.5       , -0.5       , -0.5       , ...,  0.48823529,
          0.5       ,  0.1509804 ],
        [-0.5       , -0.5       , -0.5       , ...,  0.48431373,
          0.14705883, -0.32745099],
        [-0.5       , -0.5       , -0.5       , ..., -0.32745099,
         -0.5       , -0.49607843],
        ..., 
        [-0.5       , -0.44901961,  0.1509804 , ..., -0.5       ,
         -0.5       , -0.5       ],
        [-0.49607843, -0.49607843, -0.49215686, ..., -0.5       ,
         -0.5       , -0.5       ],
        [-0.5       , -0.49607843, -0.48823529, ..., -0.5       ,
         -0.5       , -0.5       ]],

       [[-0.5       , -0.5       , -0.5       , ..., -0.5       ,
         -0.5       , -0.5       ],
        [-0.5       , -0.5       , -0.5       , ..., -0.5       ,
         -0.5       , -0.5       ],
        [-0.5       , -0.5       , -0.49607843, ..., -0.5       ,
         -0.5       , -0.5       ],
        ..., 
        [-0.5       , -0.5       , -0.5       , ..., -0.48823529,
         -0.5       , -0.5       ],
        [-0.5       , -0.5       , -0.5       , ..., -0.5       ,
         -0.5       , -0.5       ],
        [-0.5       , -0.5       , -0.5       , ..., -0.5       ,
         -0.5       , -0.5       ]]], dtype=float32)
</code></pre>

<p>I appreciate the help in advance.</p>

<p>You can find duplicates between arrays using numpy's <a href="http://docs.scipy.org/doc/numpy/reference/generated/numpy.intersect1d.html#numpy.intersect1d" rel="nofollow"><code>intersect1d</code></a> (one dimensional set intersect) function. </p>

<pre><code>duplicate_images = np.intersect1d(train_dataset, test_dataset) 
</code></pre>

<p>I timed this using the train and test sets (55000 and 10000 arrays respectively) from one of the <a href="https://www.tensorflow.org/versions/r0.10/tutorials/mnist/beginners/index.html" rel="nofollow">tensorflow tutorials</a> which I'm guessing is similar to your data. Using <code>intersect1d</code>, it took about 2.4 seconds to complete on my machine (it took only 1.3 seconds with the parameter <code>assume_unique=True</code>).  A pairwise comparison like you describe took several minutes.     </p>

<p><strong>EDIT</strong></p>

<p>This answer (above) does not compare each "image" array, as @mbhall88 points out in the comments, this compares elements within the arrays, not the arrays themselves.  In order to make sure it compares the arrays you can still use <code>intersect1d</code> but you have to mess around with the dtypes first as explained <a href="http://stackoverflow.com/a/8317403/5992438">here</a>. But, the example in that answer deals with 2d arrays and since you're working with 3d arrays you should first flatten the second two dimensions. You should be able to do something like:</p>

<pre><code>def intersect3d(A,B, assume_unique=False):
    # get the original shape of your arrays
    a1d, a2d, a3d = A.shape
    # flatten the 2nd and 3rd dimensions in your arrays
    A = A.reshape((a1d,a2d*a3d))
    B = B.reshape((len(B),a2d*a3d))
    # define a structured dtype so you can treat your arrays as single "element"
    dtype=(', '.join([str(A.dtype)]*ncols))
    # find the duplicate elements
    C = np.intersect1d(A.view(dtype), B.view(dtype), assume_unique=assume_unique)
    # reshape the result and return
    return C.view(A.dtype).reshape(-1, ncols).reshape((len(C),a2d,a3d))
</code></pre>

python arrays numpy hash