Compare elements in large list of data
<p>I've got the question on interview by python recently. The question was:</p>

<blockquote>
  <p>we have a large list of pictures in python(as I understood we simply read their content, and then got list of their contents), <code>[...]</code>, this list will occupy 1gb of RAM e.g. What is the best way to compare them(do pictures the same)?
  I answered that we can separate this list into several lists and then compare elements in them.</p>
</blockquote>

<p>But I got answer that: "it is wrong".
So my question the same what is the best way here to compare them?</p>

<p>Currently I think maybe use python's sets and compare lengths of source list and set?</p>

<p>Assuming the list contains the raw byte strings of the image contents, one quick and dirty way to weed out possible duplicates is to compare the <em>lengths</em> of the byte strings.  Two pictures with unequal length byte strings cannot be duplicates.</p>

<p>Then, for each group of pictures with equal length byte strings, hash the byte strings and compare those.  If the hashes are equal then the pictures are duplicates, otherwise they're not.  (For extra speed, don't bother hashing if there are only two byte strings in a group; just compare the strings directly byte-by-byte.)</p>

python bigdata