Checksum for a list of numbers
<p>I have a large number of lists of integers. I want to check if any of the lists are duplicates. I was thinking a good way of doing this would be to calculate a basic checksum, then only doing an element by element check if the checksums coincide. But I can't find a checksum algorithm with good properties, namely:</p>

<ul>
<li>Verifies order effectively;</li>
<li>Quick to calculate;</li>
<li>Returns a small result, eg short integer;</li>
<li>Has a fairly uniform distribution, giving a low probability of different lists coinciding.</li>
</ul>

<p>For example, a function check_sum which returned different numbers in the range [0,65536] for the following 5 calls would be ideal. </p>

<pre><code>check_sum([1,2,3,4,5])
check_sum([1,2,3,5,4])
check_sum([5,4,3,2,1])
check_sum([1,2,3,4,4])
</code></pre>

<p>I looked at the IPv4 header checksum algorithm which returns a result of about the right size but doesn't check order so isn't what I'm  looking for.</p>

<p>I'm going to implement it in python, but any format will do for algorithm, or pointer at a good reference material.</p>

<p>Calculate the checksums with <code>hash()</code>:</p>

<pre><code>checksums = \
    list(
        map(
            lambda l:
                hash(tuple(l)),
            list_of_lists
        )
    )
</code></pre>

<p>To know how many duplicates you have:</p>

<pre><code>from collections import Counter

counts = Counter(checksums)
</code></pre>

<p>To compile a unique list:</p>

<pre><code>unique_list = list(dict(zip(checksums, list_of_lists)).values())
</code></pre>

python algorithm checksum