Limits of hashes comparisons
<p>I'm storing hash codes in a file, one hash per line.</p>

<p>When I have a new hash code, I open the file and check if
the hash code already exists and if it doesn't exist,
I save it to that file.</p>

<pre><code>f = open("hashes.txt", "w")
hashes = f.readlines()
hash_code = "ff071fdf1e060400"
if not hash_code in hashes:
  hashes.append(hash_code)
for h in hashes:
  f.write(h)
f.close()
</code></pre>

<p>This code may be slow, when the number of lines in hashes.txt grows.
Is there some better storage mechanism and check to make it faster
in that case? I need fastest possible check (within a few seconds).</p>

<p>A <a href="https://docs.python.org/3.6/library/stdtypes.html#set" rel="nofollow">set</a> can be used for fast membership lookups. Since files can be used as an iterator, passing an open file handle to the set constructor will read entries into the set by line without first filling an intermediate in-memory array.</p>

<p>After this, you can simply use the set difference <code>-</code> operator to efficiently check which hashes are new and the union operator <code>|</code> to add the newly found elements to the list of known hashes:</p>

<pre><code># at program start, init list of known hashes
# open hashes_in.txt, read line by line and add to set
# set removes duplicate elements
with open("hashes.txt", "r") as f:
    hashes = set(f)

# as new hashes are encountered, use this to check if they have been seen before
def compare_hashes(search_hashes, hashes):
    search_hashes = set(search_hashes)

    # find new hashes
    new_hashes = search_hashes - hashes

    # update list of known hashes
    hashes |= new_hashes

    # write out new hashes
    with open("hashes.txt", "a") as f:
        for h in new_hashes:
            f.write(h)

    return new_hashes, hashes



with open("hashes2.txt", "r") as f:
    new_hashes, hashes = compare_hashes(f, hashes)
    print(new_hashes)
</code></pre>

<p>This answer assumes that both your list of known entries and search entries will come from files and thus have trailing newlines that will be part of the matching. If this is not what you want, you can strip newlines for a small performance overhead:</p>

<pre><code>strip_newlines = lambda hashes: (h.strip() for h in hashes)
</code></pre>

<p>Use it like so:</p>

<pre><code>hashes = set(strip_newlines(f))
new_hashes, hashes = compare_hashes(strip_newlines(f), hashes)
</code></pre>

python hash